Your e-Business Quality Partner eValid™ -- The Web Quality Suite
Browser-Based Client-Side Functional Testing and Validation Page Timing/Tuning Transaction Monitoring. WebSite Spidering & Analysis and Realistic Server Loading.

eValid -- Medical Application Monitoring Case Study
eValid Home

eValid was set up to monitor detailed intra-step real-user transaction times in a specific medical device quality monitoring role.

  1. Overview
    This case study illustrates how eValid can be used to perform a realistic test of a critical application to both confirm correct operation and to measure client-side performance.

    As part of the monitoring activity, in addition to assuring availability of the site there was a requirement to measure the elapsed time between various steps that the simulated user took.

    The outputs of the monitoring include both reporting on the eValidator monitoring site and raw timing data analyzed with a spreadsheet.

  2. Scripting Process
    Access to the application was via a secure remote connection. Using that connection we recorded scripts from the application, using the test plan supplied by the customer, and using login/password credentials supplied by the customer. (Login credentials were obfuscated by using the eValid encrypting feature, as shown in the script below).

    The test plan required the script to access live patient data from the application, requiring a complex set of steps to retrieve data from the database. We took very special care to make sure we never modified any of the data which was being shown on the screens.

    To maintain state in the application when we logged in we always logged out -- so that the next playback of the test would have identically the same steps involved.

    Once we had a recording made we played it back several times to observe the effect of adjusting the wait time multiplier. As expected, with low values of the multiplier we were able to play ahead of the application, which was built with very sophisticated AJAX coding.

    Once the de-synchronization points were identified we augmented the script by editing in different kinds of automatic playback synchronization points, e.g. SyncOnText or SyncOnElementProperty commands. To identify elements in the application page on which to base these synchronizations we used the eValid PageMap to dig into the contents of the dynamic DOM of the page. Finally, we added a series of ElapsedTime and SaveRecord commands to extract data about performance of the individual steps (see script below.)

  3. Performance Data Analysis
    We put the script in "monitoring mode" by scheduling it to run on 15-minute intervals using the Windows scheduler and the eValid Batch Interface. The generated EventLog was fed by a remote connect into the eValidator reporting engine, and the accumulated raw timing data generated by the SaveRecord commands was emailed back to a special email address for processing locally. The raw timing data was pre-formatted in the script to be ready to drop into a spreadsheet.

    Except for the period when the eValid playback was active there is no other interference on the monitoring machine.

  4. Monitoring Reports
    We used the NagVis monitoring reporting system to show the results that were collected as the tests run. The screenshot below shows response time history for a 1-month period.

    Fully santifized and version of actual 
eValid monitoring output data as seen from
the eValidator monitoring reporting portal.
    (Click for full sized image)

  5. Offline Analysis Reports
    Here is a sample of a 1-day period of collection of data. In the columns you see the individual times, expressed in milliseconds, for each of the several internal intervals being measured.

    Table of timing data, expressed in milliseconds,
collected from the script described on this page, based 
on performance as measured from Windows client running 
eValid and accessing the target medical analysis/reporing 
application over a real-world web connection.
    (Click for full sized image)

    After analysis of the data, you can see how this looks in this graphic which shows various timed intervals as a function of elapsed time.

    Spreadsheet produced graph of raw timing data, 
measured in milliseconds, from the above table showing 
variation in measured response time parameters as 
a function of time, for a single measurement period.
    (Click for full sized image)

  6. Monitoring Script
    This script makes use of structural commands to support automated login, and makes use of eValid DOM-sync commands to assure playback synchronization. (This script has been modified slightly to assure privacy of named individuals and companies.)

    # Recording by eValid V9 
    # (c) Copyright by Software Research, Inc. 
    ProjectID "eValidTests"
    GroupID "Scripts"
    TestID "CaseStudy"
    InitLink "http://APPLICATION-UNDER-TEST/default.aspx"
    SyncOnElementProperty 0 "id" "imgTracker" ""
    IndexSet 0
    IndexFindElement 0 DOWN "id" "imgFD" ""
    IndexFollowLink 0 ""
    ElapsedTime "01_Start"
    SaveRecord "Date: %T\t01: %e"
    # ----------------------------------------------------------
    SyncOnText 0 "Username : " ""
    SyncOnElementProperty 0 "id" "btnSign" ""
    ElementFocus 0 50 "txtUsername" "" ""
    # Wait 3016
    # clLClick 0 706 361 0
    Wait 2531
    # InputValue 0 50 "TEXT" "txtUsername" "txtUsername" "«î‰ç®Š»" "" ""
    IndexSet 0
    IndexFindElement 0 DOWN "id" "txtUsername" ""
    IndexInputValue 0 "TEXT" "«î‰ç®Š»" "txtUsername" ""
    Wait 2390
    # InputValue 0 57 "TEXT" "txtPassword" "txtPassword" "«³p¡ô‰ö¡„»" "" ""
    IndexSet 0
    IndexFindElement 0 DOWN "id" "txtPassword" ""
    IndexInputValue 0 "TEXT" "«³p¡ô‰ö¡„»" "txtPassword" ""
    Wait 1000
    # Recorded script passage below replaced with DOM commands...
    IndexSet 0
    IndexFindElement 0 DOWN "id" "btnSign" ""
    IndexElementClick 0 ""
    # InputImageClick 0 34 "http://APPLICATION-UNDER-TEST/check.btn" '' ''
    ElapsedTime "02_Treatment_Op"
    SaveRecord "\t02: %e"
    # ----------------------------------------------------------
    SyncOnElementProperty 0 "id" "lblLoadingMessage" ""
    Wait 1000
    SyncNotOnElementProperty 0 "id" "lblLoadingMessage" ""
    SyncOnElementProperty 0 "id" "MainContent_Shiftupdatepnl" ""
    SyncOnElementProperty 0 "id" "MainContent_gvpatientdashboard_hlinkltd3_1" ""
    Wait 2156
    # GotoLink 0 "http://localhost/APPLICATION-UNDER-TEST FD/Dashboard" "" 
    Wait 3578
    # Recorded script passage below replaced with DOM commands...
    IndexSet 0
    IndexFindElement 0 DOWN "innerText" "Patient, Name" ""
    IndexMove +1
    Wait 100
    IndexFollowLink 0 ""
    # FollowLink 0 561 "Patient, Name" "javascript:__doPostBack" ""
    ElapsedTime "03_HomeAnalysis"
    SaveRecord "\t\t03: %e"
    # ----------------------------------------------------------
    Wait 1000
    SyncOnText 0 "Analysis Adequacy" "number:2"
    Wait 2344
    ValidateSelectedText 0 117 0 "Analysis Adequacy" "number:2"
    Wait 1047
    # FollowLink 0 124 "Analysis Adequacy" "javascript:__doPostBack('TreePOCCo" \
    #	"ndition','sInActive')" "number:2"
    FollowLink 0 121 "" "javascript:TreeView_ToggleNode(TreePOCCondition_Data,4,TreePOCConditionn" \
    	"4,'%20',TreePOCConditionn4Nodes)" "number:2" NO_NAV
    Wait 1484
    SyncOnText 0 " Std Kdt/V in goal range" "number:2"
    Wait 2859
    ValidateSelectedText 0 126 17 " Std Kdt/V in goal range" "number:2"
    # Add the time for actual LOGOUT...
    Wait 3499
    # Recorded script passage below replaced with DOM commands...
    IndexSet 0
    IndexFindElement 0 DOWN "id" "BTNLogout" ""
    IndexFollowLink 0 ""
    # FollowLink 0 86 "Worklist" "javascript:__doPostBack('ctl00%24ctl00%24" \
    #	"MainMenu','Worklist')" ""
    SyncOnText 0 "Username : " ""
    Wait 1047
    # FollowLink 0 141 "Facility" "http://localhost/APPLICATION-UNDER-TEST/" \
    #	"Dashboard/DailyTreatment.aspx" ""
    ElapsedTime "04_Logout"
    SaveRecord "\t\t\t\t05: %e\t TOTAL: %E"
    # ----------------------------------------------------------
    SendFile "" "CASE-STUDY.evs playback results" ""
    # End of script.