The following describes the methodology applied to some of the data transfer tests we are performing for various cloud storage platforms. In each case, the following approach should be assumed with the exception of test-specific details which will be posted with each result set.
Disclaimer:
- The research team understands that any time the public internet is introduced into a test a number of non-controllable factors are introduced. It is the intent of this project to test various scenarios often enough and with enough variance to obtain a reasonable average and thereby allow the team to make general assumptions about the quality of service (given the constraints stated) that one can reasonably expect to encounter when utilizing a given service.
- It is similarly understood that there may exist environmental factors (i.e. routing paths, proxy servers, firewalls) that affect the transfer rates being tested. In general, it is believed that these factors should affect all tested platforms equally. However, in the case of various research institutes where specialty networks (i.e. ESNet, NLR, I2) exist, there may be routing configurations that particularly favor one service or endpoint over another. It is an objective of these tests to expose these anomalies with the goal of addressing them as appropriate.
- Baseline: For the various services tested, these initial tests were performed using no particular optimization techniques. We took the respective vendor’s shipping SDK, integrated it into a very similar wrapper (source code available for verification) and executed it. Subsequent work should focus on optimizations in the SDKs, or the methods in which the libraries are utilized, etc.
- Not A Stand-Alone Work: This data should not be considered in isolation. Rather, it is a portion of a larger data set (some of which may remain to be published) and should be interpreted for what it is – a portion of a larger collection that aims to provide a more complete view of the entire problem domain.
Test Objectives:
- General: Generate data to set expectations for users of various cloud services focusing on a scenario of local compute combined with cloud-hosted data (blob storage). Note: the reverse scenario as well as cloud-hosted compute/cloud-hosted data will be tested separately
- These tests and data are crucial to our overall objective of improving the experience of researchers interacting with cloud computing assets as they provide a baseline against which any optimizations or alterations may be compared.
Test Setup:
- Test Setup
- A collection of random-data files were generated (RandomFileGenerator.exe). For each of the following file sizes, 50 files were generated and stored on standard disks local to the test computer: Range is specific to each test set.
- Network Connectivity: specific to each test set
Test Execution:
- For each file size, AWS_Console_App1.exe was called to upload the files to Amazon’s US Standard Region and record the duration
.\amazon\aws_console_app1.exe .\data\2KB
- For each file size, DownloadFiles.exe was called to download the files just uploaded to Amazon’s US Standard Region and record the duration
.\downloader\DownloadFiles.exe -i .\amazon_2KB.csv -p 6 -m yes
- For each file size, AzureTesting.exe was called to upload the files to Azure’s US North Central region and record the duration
.\azure\azuretesting.exe .\data\2KB
- For each file size, DownloadFiles.exe was called to download the files just uploaded to Azure’s North Central region and record the duration
.\downloader\DownloadFiles.exe -i .\azure_2KB.csv -p 6 -m yes
- NOTE: immediately following each operation for each file size, the resulting file (log.csv) was renamed to represent the source, transfer direction, and file size
ren log.csv azure_ussc_upload_2KB.csv
Report Generation:
- For each service tested, and each file size tested
- For both Uploads and Downloads (separately)
- Scatter plot is generated showing the distribution for the transfer duration (seconds)
- Scatter plot is generated showing the distribution for the transfer rate (Mb/s)
- Transfer duration average (seconds) is calculated
- Transfer duration standard deviation (seconds) is calculated
- Transfer rate average (Mb/s) is calculated
- Transfer rate standard deviation (Mb/s) is calculated
- For each file size tested
- For both Uploads and Downloads (separately)
- A comparison chart (column) is generated showing the average transfer duration (seconds) and error bars indicating one standard deviation (seconds). Also plotted is a dot indicating the associated average transfer rate on the secondary Y axis (Mb/s)
- Summary Charts
- For both Uploads and Downloads (separately)
- A range chart is generated showing the band covered by one standard deviation (per service tested) for the transfer duration (seconds) across the tested file sizes
- A range chart is generated showing the band covered by one standard deviation (per service tested) for the transfer rate (Mb/s) across the tested file sizes
- Presentation
- Once the above charts have been generated, they are assembled into a PowerPoint file
- Once the power point file has been generated and saved, it is published as a PDF file
- Automation
- All of the above steps are automated via a script (ProcessTransferLogs.ps1)
Conventions:
- Naming Conventions
- Amazon_USSTD: Amazon’s US Standard region was specified when the bucket was created
- Azure_USNC: Azure’s US North Central region was selected when the storage account was created
- Error Handling
- In most runs, errors were displayed to the screen but not captured to logs.
- Existence of errors (all of which were network-related) are manifested in the logs as collections of data points less than 50 (the test source size)
- Due to the fact that the respective download tests are based on the upload source files, a download file containing less than 50 entries is not necessarily indicative of errors but may simply be tied to the fact that the input file had less than 50 entries. This being said, there were more errors on downloads than uploads.
Resources:
Results: Specific to each test set
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
[NOTE: these are simply introductory tests and should not be considered scientifically accurate comparisons of the various platforms]
The second run of the tests used the defaults for the parameters as provided in the original sample with the exception of the “runs” parameter which was increased from 1,000,000 to 10,000,000. The test bed used is detailed here and the results of the first run are here. The net-net is that the calculation is run 100 times and each time contains a loop that runs 10,000,000 times. Each run of the calculation can be run independently of the others as the aggregation/summation is handled in the spreadsheet once the calculations have finished. To provide some protection from anomalies, I ran each test against each platform 10 times with the first run being “cold”.
Results:
The time values on the Y axis are seconds to complete the execution. In this run, a few things jump out:
- As in the first test, the 2-node HPC cluster killed the other two. Again this isn’t surprising, although I’d like to think that as the compute to communication ratio grows, the delta b/t the HPC cluster and Azure will shrink.
- In contrast to the first run, in this run Azure beat the local run by an average of nearly 200 seconds/run. This is consistent with the theory that the communication overhead in the Azure solution is constant per request regardless of the complexity of compute resulting with the communication overhead becoming marginalized as the compute to communicate ratio increases.
I’m currently performing tests that take the runs parameter to 100,000,000 which should separate the three platforms further and it is expected that unlike the first two runs, Azure will be closer to the HPC results than to the local compute results (although the HPC cluster is still expected to outperform Azure).
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
[NOTE: these are simply introductory tests and should not be considered scientifically accurate comparisons of the various platforms]
The first run of the tests used the defaults for the parameters as provided in the original sample. The test bed used is detailed here. The net-net is that the calculation is run 100 times and each time contains a loop that runs 1,000,000 times. Each run of the calculation can be run independently of the others as the aggregation/summation is handled in the spreadsheet once the calculations have finished. To provide some protection from anomalies, I ran each test against each platform 10 times with the first run being “cold”.
Results:
The time values on the Y axis are seconds to complete the execution. In this run, a few things jump out:
- The 2-node HPC cluster killed the other two. This is not altogether surprising, especially considering the processor power is significantly better than the other two scenarios and this is the only scenario where the compute is happening on raw hardware (the “local” development machine is virtualized and I believe that the node instances hosted in Azure are also virtualized.
- The HPC cluster took one run to “warm up” after which the performance was remarkably solid.
- The “Local” (sequential) run beat the Azure (parallel) run. At first this might be surprising, but it really isn’t when you think about it. The Azure model has a certain communication overhead for each request (HTTP sending of each request to azure, communication b/t the web and worker roles via the queue, storage of results in the Azure tables, and the retrieval of the results). In this test run, the compute per run is not significant (averaging just over 0.5 seconds on the local box per calculation) and the communication overhead stands out more. It is supposed that in subsequent runs where the compute is heavier, the two will trade places.
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
I’m working through some basic performance testing and comparisons for cloud compute and this post is intended to provide a bit of detail regarding the test platform I’m using for conducting these tests. I should also be clear that the test platforms below are not designed to be perfect-world hardware platforms, but rather proof-of-concept and suitable for order-of-magnitude testing.
General Code Base
For this aspect of testing I have been working with the Asian Options Pricing WCF sample provided by Microsoft on the HPC resource kit site (http://resourcekit.windowshpc.net/). This is a simple VSTO-enabled worksheet that calculates the price of an option on the Asian market. I’m by no means a market analyst nor can a vouch for the accuracy of the calculation, but it does provide a CPU-intensive operation that can be parallelized easily making it a good candidate for scale-out compute. The sample is designed to be illustrative of submitting WCF jobs (‘micro jobs’) to a Windows HPC cluster and comes with source and instructions for getting it setup and running.
In the default configuration, the worksheet performs 100 iterations of Monte Carlo pricing runs using a function called PriceAsianOptions (code listed at the bottom of this post). The basic theory may be understood better by reading this article. The function takes the following input parameters:
- Up – the specific factor by which the underlying instrument may move up per step of the binomial price tree
- Down – the specific factor by which the underlying instrument may move down per step of the binomial price tree.
- Interest – the continuously compounded, risk-free interest rate. This number doesn’t affect the complexity of the calculation so I left it constant.
- Initial Price – the starting or current price of the stock. I left this constant as it doesn’t affect the complexity of the calculation
- Periods – the number of periods (trading days) before the option expires. The default was 20 and I left this constant. Technically it could increase the complexity of the calculation but changing the value of the Runs parameter does a suitable job of this.
- Exercise – the price at which the call should be exercised
- Runs – within a given calculation, how many times should the calculation be done prior to averaging and returning the results. This defaulted to 1,000,000 and is the prime value that I altered during the runs to increase the duration of the calculation.
Once the worksheet has finished calculating, a batch (by default 100 such calls to PriceAsianOptions) it calculates and displays the Average of the Monte Carlo runs, the Min, Max, Standard Deviation, Standard Error, and Execution Time in seconds.
NOTE: this test platform is much less about the specific problem being solved and more about taking a CPU/Compute intensive problem and expressing it on different platforms.
Windows HPC Server Environment
Hardware: My test environment consists of a small cluster with one head node and two compute nodes. The head node is a single-proc box running with 2 GB of RAM and 2 NICs. Besides the cluster Head Node role, it is also running DNS, DHCP, AD, SQL and has the WCF broker role (it does *not* have the compute role). Each compute node is a dual core Intel box with 4 GB RAM running Windows 2008 HPC Server. The cluster is configured such that the head node has one leg on the “enterprise network” (in my corner of the universe this is simply my lab network) and another leg on the private network shared with the compute nodes. The individual compute nodes are not accessible from outside this private network.
Software: While the instructions and download would lead you to believe that the code is ready to run out-of-the-box (OOTB) that is not quite true. Beyond the changes detailed in the instructions, I had to make some changes in additional locations for the name of the cluster’s head node as well as that to the dll as seen from the compute nodes. Beyond those changes however, the code used in the tests is identical to what is available on the resource kit site. The main logic is a loop to submit n jobs, with each request having an asynchronous event to process the results.
Local Compute Environment
Hardware: The platform I’m using for the “local machine” compute tests is a Windows Server 2008 Standard box (64 bit) with 4 GB of RAM and a single 2.4 Ghz processor. This certainly isn’t anything fancy but should provide a middle-of-the-road hardware spec for comparison to the other platforms.
Software: I started by taking the worksheet used for the HPC environment and added another button for the “local” runs. Since the hardware I’m running on only has a single processor, I adjusted the logic loop to not be parallelized (additional threads would simply harm performance) but rather sequential. The work is performed on a background thread and as soon as an individual computation is completed the UI is notified/updated.
Azure Environment
Hardware: The configuration for the test bed as hosted in Azure is split over two projects. The first project is the data project (used for queues and tables) and is not part of any affinity group and is configured for a geographic location of USA-Anywhere. The second project is the compute project and is configured with one web role and two worker roles. It is also not part of an affinity group and is configured with a geographic location of USA-Anywhere. NOTE: the lack of specifically selecting an affinity group or geographic location is to provide a sort of worst-case scenario assuming that selecting either of those would, if anything, only improve the performance.
Software: There was a good bit more coding to do here as I attempted to mimic (in theoretical approach at least) the general implementation of WCF services on Windows HPC. The web role is host to a WCF service that allows a client (in this case the Excel worksheet) to submit a single pricing request (providing the parameters explained above). This pricing request and parameters is serialized and placed on the Azure queue to be picked up by one of the running worker roles (incidentally, one nice side affect of this approach is that from a coding standpoint it makes no difference how many worker roles exist – if more are needed they can simply be configured and items will be processed off the queue in a quicker fashion). Once the worker role picks up the pricing request/data, it processes the request and places the resulting price (and a the request identifier) into a row in an Azure table. At this point it checks the queue again for the next pricing request. An additional method exists in the WCF service that accepts a request ID and then looks at the azure table for a result matching that request ID. If one is found, the result is returned and the data removed from the azure table.
On the Excel worksheet side, I modified the same worksheet as before with yet another button for submitting the work to the Azure-hosted WCF service. I took a similar loop to the other two scenarios but made the adjustment that it will first submit all of the requests and only then will it begin asking for the results. The idea is to avoid having the requests for results clogging the network and slowing down the submission of jobs – I wanted to keep the worker roles as busy as possible until the work was done. I initially implemented this using a similar approach to the HPC code (async anonymous methods handling the results) but this failed as the tests grew bigger because of the default web server timeouts (i.e. I’d have 100 requests queued and it was likely that more than 90 seconds would pass before the results were finished).
Sample Code
private double PriceAsianOptions(double initial,
double exercise, double up, double down, double interest,
int periods, int runs)
{
double[] pricePath = new double[periods + 1];
// Risk-neutral probabilities
double piup = (interest - down) / (up - down);
double pidown = 1 - piup;
double temp = 0.0;
Random rand = new Random();
double priceAverage = 0.0;
double callPayOff = 0.0;
for (int index = 0; index < runs; index++)
{
// Generate Path
double sumPricePath = initial;
for (int i = 1; i <= periods; i++)
{
pricePath[0] = initial;
double rn = rand.NextDouble();
if (rn > pidown)
{
pricePath[i] = pricePath[i - 1] * up;
}
else
{
pricePath[i] = pricePath[i - 1] * down;
}
sumPricePath += pricePath[i];
}
priceAverage = sumPricePath / (periods + 1);
callPayOff = Math.Max(priceAverage - exercise, 0);
temp += callPayOff;
}
return (temp / Math.Pow(interest, periods)) / runs;
}
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
I’ve been thinking quite a bit lately about the role of cloud computing as it applies to scientific research (as hinted at by the title of this site). One possible flaw in my approach is that I’ve been delving into MPI-based compute as much as I can to wrap my mind around how it works with the notion of then applying that paradigm to cloud compute. I list it as a flaw only because I wonder if it is possibly time to think a bit further outside the proverbial box if you will. I’ve been mulling over the following:
How do people actually use multiple machines to solve a problem? – This is really the root question behind all of this work. The first scenario is high-end shared-memory machines (ala Cray supercomputers) and I’m going to eliminate that type of compute from the conversation due to the fact that it simply can’t be well-replicated in the cloud as we currently know it. The far opposite end of the spectrum is “manual” clustering or map reduce – someone figures out a problem they want to solve, divvy’s it up amongst N nodes, and then individually runs a program on each node with the appropriate settings and then manually aggregates the results. This extreme is most likely done by ad-hoc projects or those not familiar with traditional HPC technologies and approaches. Between the two extremes listed, there are Map/Reduce implementations and traditional MPI programs targeted at distributed memory systems.
Amazon’s EC2 – very easy to utilize for lower-throughput MPI-based HPC - given you can get n Linux boxes for 0.10/cpu/hour and, because of the vast community that has grown up around it, there are pre-packaged clusters (via AIM) and even commercial vendors building businesses on top of providing HPC-style compute in EC2 in an “on-demand” fashion. Further, traditional grid computing platforms such as Nimbus have been radically adapted to provide a rather compelling local – to – cloud story for scientific HPC. It would seem that if you are working in HPC today, and simply want to utilize an HPC cluster “in the cloud” (maybe because of lack of access to sufficient hardware) that Amazon’s EC2 and the toolsets such as Nimbus (and others) that sit on top of it is a natural solution.
Microsoft’s Azure – While it is a quickly adapting platform (seeing as it hasn’t yet released) and they have hinted at plans to adapt the platform based on customer demand, if you look at it currently, there’s not an obvious fit for the traditional HPC model. The customer of Azure is given the choice of deploying web or worker roles, and one can imagine using worker roles in a fashion analogous to cluster nodes… but there currently isn’t any built-in infrastructure to bind those nodes into a single group/cluster. As it stands now, Azure seems to lean towards the manual-approach to large-scale compute. What could change this story completely is if Microsoft decided to offer HPC Pack-enabled nodes as a type of resource you could request, although there’s been nothing to hint that they are planning anything like this.
Where do we go next? – I’ve been chewing on whether or not it makes any sense to try to push HPC-style work into Azure, or if it should simply be relegated to the EC2’s of the world… One could conceivably build an implementation of MPI that, rather than relying on the underlying cluster would provide cloud-style/enabled communications between nodes… this could allow those most comfortable with (or with large existing code bases of) MPI-style apps to continue to utilize those libraries/applications, but one has to wonder if, unless the Microsoft pricing (to be announced later this summer) is incredibly cheaper than that of EC2, why would one bother (other than academic interest, of course) to build such? Again, this could be mitigated by Microsoft providing such itself, but the platform would have to provide additional compelling aspects to pull someone away from what would otherwise be a very comfortable transition (local cluster to an EC2-hosted cluster running the same software stacks).
What I’ve really been wondering – is if it is not time to throw MPI out altogether (or, more accurately, the programming paradigm that it represents). Is it time to look for ways to raise the level of abstraction for the computational researcher… and, if so, does something like Azure have a more interesting role? I’m wondering if some of the abstraction tools (workflow engines, queue services, etc.) will begin to have a role or if we need to continue to stretch for every raw bit of horsepower from the system (acquiescing to the fact that abstraction layers cost is in reduced raw system power). For many of the large simulation models it seems that the raw horsepower is indeed necessary. You also cannot simply ignore the vast collection of existing tools and libraries that already exist and target this paradigm. The flip question is that is there a collection of computational research for which, if the cost per cpu hour was low enough, and the increase in development productivity was great enough (assuming that the proposed layers of abstraction resulted in such), would it really matter if the job took an additional 30-50% time to run? This is, of course, only salient if we live in a world wherein I can get however many compute nodes I want whenever I want them (no waiting in queue).
My gut tells me we aren’t quite there yet, but I wonder how far out it really is?
Currently rated 5.0 by 1 people
- Currently 5/5 Stars.
- 1
- 2
- 3
- 4
- 5