eValid™ -- The Web Quality Suite Browser-Based Client-Side Functional Testing and Validation Page Timing/Tuning Transaction Monitoring. WebSite Spidering & Analysis and Realistic Server Loading.

Synopsis
eValid was set up to monitor detailed intra-step real-user transaction times in a specific medical device quality monitoring role.

1. Overview
   This case study illustrates how eValid can be used to perform a realistic test of a critical application to both confirm correct operation and to measure client-side performance.

   As part of the monitoring activity, in addition to assuring availability of the site there was a requirement to measure the elapsed time between various steps that the simulated user took.

   The outputs of the monitoring include both reporting on the eValidator monitoring site and raw timing data analyzed with a spreadsheet.

2. Scripting Process
   Access to the application was via a secure remote connection. Using that connection we recorded scripts from the application, using the test plan supplied by the customer, and using login/password credentials supplied by the customer. (Login credentials were obfuscated by using the eValid encrypting feature, as shown in the script below).

   The test plan required the script to access live patient data from the application, requiring a complex set of steps to retrieve data from the database. We took very special care to make sure we never modified any of the data which was being shown on the screens.

   To maintain state in the application when we logged in we always logged out -- so that the next playback of the test would have identically the same steps involved.

   Once we had a recording made we played it back several times to observe the effect of adjusting the wait time multiplier. As expected, with low values of the multiplier we were able to play ahead of the application, which was built with very sophisticated AJAX coding.

   Once the de-synchronization points were identified we augmented the script by editing in different kinds of automatic playback synchronization points, e.g. SyncOnText or SyncOnElementProperty commands. To identify elements in the application page on which to base these synchronizations we used the eValid PageMap to dig into the contents of the dynamic DOM of the page. Finally, we added a series of ElapsedTime and SaveRecord commands to extract data about performance of the individual steps (see script below.)

3. Performance Data Analysis
   We put the script in "monitoring mode" by scheduling it to run on 15-minute intervals using the Windows scheduler and the eValid Batch Interface. The generated EventLog was fed by a remote connect into the eValidator reporting engine, and the accumulated raw timing data generated by the SaveRecord commands was emailed back to a special email address for processing locally. The raw timing data was pre-formatted in the script to be ready to drop into a spreadsheet.

   Except for the period when the eValid playback was active there is no other interference on the monitoring machine.

4. Monitoring Reports
   We used the NagVis monitoring reporting system to show the results that were collected as the tests run. The screenshot below shows response time history for a 1-month period.
   
   (Click for full sized image)

5. Offline Analysis Reports
   Here is a sample of a 1-day period of collection of data. In the columns you see the individual times, expressed in milliseconds, for each of the several internal intervals being measured.
   
   (Click for full sized image)

   After analysis of the data, you can see how this looks in this graphic which shows various timed intervals as a function of elapsed time.

   
   (Click for full sized image)

6. Monitoring Script
   This script makes use of structural commands to support automated login, and makes use of eValid DOM-sync commands to assure playback synchronization. (This script has been modified slightly to assure privacy of named individuals and companies.)

   # Recording by eValid V9 # (c) Copyright by Software Research, Inc. ProjectID "eValidTests" GroupID "Scripts" TestID "CaseStudy" DeleteCache DeleteCookies ResetTimer InitLink "http://APPLICATION-UNDER-TEST/default.aspx" SyncOnElementProperty 0 "id" "imgTracker" "" IndexSet 0 IndexFindElement 0 DOWN "id" "imgFD" "" IndexFollowLink 0 "" ElapsedTime "01_Start" SaveRecord CASE-STUDY.data.txt "Date: %T\t01: %e" # ---------------------------------------------------------- SyncOnText 0 "Username : " "" SyncOnElementProperty 0 "id" "btnSign" "" ElementFocus 0 50 "txtUsername" "" "" # Wait 3016 # clLClick 0 706 361 0 Wait 2531 # InputValue 0 50 "TEXT" "txtUsername" "txtUsername" "«î‰ç®Š»" "" "" IndexSet 0 IndexFindElement 0 DOWN "id" "txtUsername" "" IndexInputValue 0 "TEXT" "«î‰ç®Š»" "txtUsername" "" Wait 2390 # InputValue 0 57 "TEXT" "txtPassword" "txtPassword" "«³p¡ô‰ö¡„»" "" "" IndexSet 0 IndexFindElement 0 DOWN "id" "txtPassword" "" IndexInputValue 0 "TEXT" "«³p¡ô‰ö¡„»" "txtPassword" "" Wait 1000 # Recorded script passage below replaced with DOM commands... IndexSet 0 IndexFindElement 0 DOWN "id" "btnSign" "" IndexElementClick 0 "" # InputImageClick 0 34 "http://APPLICATION-UNDER-TEST/check.btn" '' '' ElapsedTime "02_Treatment_Op" SaveRecord CASE-STUDY.data.txt "\t02: %e" # ---------------------------------------------------------- SyncOnElementProperty 0 "id" "lblLoadingMessage" "" Wait 1000 SyncNotOnElementProperty 0 "id" "lblLoadingMessage" "" SyncOnElementProperty 0 "id" "MainContent_Shiftupdatepnl" "" SyncOnElementProperty 0 "id" "MainContent_gvpatientdashboard_hlinkltd3_1" "" Wait 2156 # GotoLink 0 "http://localhost/APPLICATION-UNDER-TEST FD/Dashboard" "" Wait 3578 # Recorded script passage below replaced with DOM commands... IndexSet 0 IndexFindElement 0 DOWN "innerText" "Patient, Name" "" IndexMove +1 Wait 100 IndexFollowLink 0 "" # FollowLink 0 561 "Patient, Name" "javascript:__doPostBack" "" ElapsedTime "03_HomeAnalysis" SaveRecord CASE-STUDY.data.txt "\t\t03: %e" # ---------------------------------------------------------- Wait 1000 SyncOnText 0 "Analysis Adequacy" "number:2" Wait 2344 ValidateSelectedText 0 117 0 "Analysis Adequacy" "number:2" Wait 1047 # FollowLink 0 124 "Analysis Adequacy" "javascript:__doPostBack('TreePOCCo" \ # "ndition','sInActive')" "number:2" FollowLink 0 121 "" "javascript:TreeView_ToggleNode(TreePOCCondition_Data,4,TreePOCConditionn" \ "4,'%20',TreePOCConditionn4Nodes)" "number:2" NO_NAV Wait 1484 SyncOnText 0 " Std Kdt/V in goal range" "number:2" Wait 2859 ValidateSelectedText 0 126 17 " Std Kdt/V in goal range" "number:2" # Add the time for actual LOGOUT... Wait 3499 # Recorded script passage below replaced with DOM commands... IndexSet 0 IndexFindElement 0 DOWN "id" "BTNLogout" "" IndexFollowLink 0 "" # FollowLink 0 86 "Worklist" "javascript:__doPostBack('ctl00%24ctl00%24" \ # "MainMenu','Worklist')" "" SyncOnText 0 "Username : " "" Wait 1047 # FollowLink 0 141 "Facility" "http://localhost/APPLICATION-UNDER-TEST/" \ # "Dashboard/DailyTreatment.aspx" "" ElapsedTime "04_Logout" SaveRecord CASE-STUDY.data.txt "\t\t\t\t05: %e\t TOTAL: %E" # ---------------------------------------------------------- SendFile "data@e-valid.com" "CASE-STUDY.evs playback results" "CASE-STUDY.data.txt" # End of script.