eValid™ -- Automated Web Quality Solution Browser-Based, Client-Side, Functional Testing & Validation, Load & Performance Tuning, Page Timing, Website Analysis, and Rich Internet Application Monitoring.

Summary
Using eValid LoadTest capability we ran a series of experiments that involved driving a mobile web application in ramped loading scenarios, up to 1,000 BUs in parallel. The results we generated strongly suggest that a performance limitation is an easily recognizable bottleneck in the infrastructure of the mobile application web server stack.

Chart of Derived Internal Response Times (Click to See Full Image)

Conclusions About Capacity
The chart graphs the basic page download time of the retrieval of train departure details into the mobile device, as a function of the total number of simultaneous users, which grows linearly over the 30-minute scenario duration. This chart suggests that the server-stack I/O capacity has been reached. You can see on the curve that the delays begin to build at a level of about 150 BUs.

Working backward and assuming linearity in the driven load, we estimate that the actual I/O channel capacity of the application server is 10 Mbps.

Technical Details
Here are the technical details about how this experiment was done.

• UserAgentString Setting
  The script sets evalid to the iPhone UserAgentString (UAS) as follows:

  "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+
      (KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3"

• LoadTest Ramp Schedule
  The ramp scenario is 1[1]1,000 BUs at the rate of one new BU per second, so the ramp up required just over 1,000 seconds = 0.27 hours = 16.67 minutes. To do this we used 10 separate user desktops on one machine, allocated at 100 BUs per desktop.

• Driver Machine Details
  For this experiment we used a machine image on the Amazon Compute Cloud. The driver machine was an Amazon EC2 "m2.4xlarge" instance, which has 26 ECUs (Equivalent Computing Units), 8 cores, 68.4 GBytes RAM, and I/O bandwidth = 1 Gbps (1,000 Mbps).

  During the experiment we observed the percentage of CPU utilization when all 1,000 BUs were running was ~45%. At the same time, the percentage of RAM used by all 1,000 BUs was ~20%.

• Response Time Data Collection
  The SaveRecord in the script (see below) recorded the important wait time between making a request for data and the final delivery of the data. A total of 5,910 completed playbacks each yielded one of the data points shownin the chart (above).

• Test Download Volume
  We measured the total amount of material downloaded by a singleton playback as 443 KBytes. This measurement was done by running the script one time, while noting the total Bytes Downloaded in the test summary page.

• Steady State Driver I/O Channel Usage
  In steady state, with 1,000 BUs running in parallel, each BU implies 443 KB/cycle downloaded (singleton data volume), completing at ~325 secs/cycle.

  The channel on the driver machine is 1,000 Mbps, so this I/O load is only ~10% of the driver machine channel capacity.

• Driver Machine Desktop Screenshot
  Here is a screen view of the driver computer with 10 users' desktops operating in parallel. If you look carefully you will see that each of the user desktops shows ≥ 100 BUs running per desktop, for a total of ≥ 1,000 BUs overall.

  
  Click For Full-Size Image

  
  

Playback Script
The playback script used in this test is shown below. The actual product URL and some of the script details have been disguised.

# Recording by eValid V9 # Copyright © 2012 by Software Research, Inc. # Recording made on: Microsoft Windows Vista ProjectID "LoadTests" GroupID "MobileDemos" TestID "Mobile-1000" LogID "AUTO" ScreenSize 1280 1024 FontSize 0 DeleteCache DeleteCookies # Navigate to a webpage that contains any link like "Mobile XXXXXXXXXX" # which spawns/redirects to a new target SUB-WINDOW (titled for example # "Mobile Web APPLICATION") instance containing your href... # InitLink "http://www.myMobileSite.com" Wait 3000 # Set up a sub-window for Apple iPhone 4 emulation... SetUserAgent "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+" \ " (KHTML, like Gecko) Version/3.0 Mobile/1C25 Safari/419.3" Wait 2236 # This sequence of INDEX MOTION commands will allow for the re-assignment # of varying target=SUB-WINDOW titled instances, else we are limited to # the same referenced target for each separate parent browser spawned... # IndexSet 0 IndexFindElement 0 DOWN "innerText" "Mobile XXXXXXXXXX" "" ValueSet "_window_$USER_$NUMB" ValuePutElement 0 "target" "" IndexFollowLink 0 "" # The subwindow now will report data while dressed as the specified device... SubFrameSet 1 "Mobile Web APPLICATION" Wait 5192 # -------- Key Performance Indicator (KPI) Timing steps --------------------- ResetTimer FollowLink 1 24 "Real Time Departures" "http://XXXXXXXXXX/schedules/eta/" \ "index.aspx" "" SyncOnText 1 "Departures as of" "" SyncOnText 1 "min" "" SyncOnText 1 " car)" "" # Click internal stopwatch and save data for chart... ElapsedTime "Train Schedule" SaveRecord DownloadTime.txt "%T \tInstance: $USER_$NUMB_$_RunCount\tTime: %E" # --------------------------------------------------------------------------- # Wait a little bit (0.5 sec) to assure a clean exit from the playback. Wait 500