Performance Testing: Open Source Approach



Performance Testing: Open Source Approach

Introduction:-

For any web application, the end to end response time for a user is a critical issue. As the number of users accessing the same web application increases, the performance of the application decreases due this concurrent user load on the server. To ensure business continuity, organizations often want to know and understand the application behavior under projected user load before it goes live. The challenges of successfully conducting an online business are as follows:

1) High User Volumes

2) Low response times for web requests

3) The overall reliability of Service

Existing Options for Performance Testing

Some popular commercial tools available for performance testing and diagnostics are

i) Mercury Load Runner

ii) IBM Rational Performance Tester

iii) IBM Rational Robot

iv) Borland Silk Performer

These tools in general provide user with excellent GUI options to record the transaction between client and server. These tools also provide enough flexibility to modify the scripts in order to simulate the dynamic data passed between client and server. User can then create user\business scenarios to generate load on the server and monitor the different aspect of server response like response time, resource utilization etc.

The commercial tools also provide tuning consoles that can gather the data from different tiers of the application and present an integrated view of performance of these layers\tiers under load.

Popular Open source Performance Testing Solutions

Though there are many open source solutions available under GNU that provide an alternate solutions to commercial tools, for the purpose of this paper, we have selected the most popular tools for performance testing of web (HTTP\HTTPS) applications.

i) OpenSTA

ii) Apache JMETER

iii) WebLoad

Open Source Tools vs. Commercial tools

Practically, Open source tools lag from their commercial counterparts in the following areas

▪ Effective Integrated Monitoring and analysis capability

▪ User network speed simulation

▪ IP spoofing capability

The Practical Approach to use Open Source Load Testing Tools

The open source tools can be cost effective substitution for commercial tools to load test your web application, provided the nooks are well taken care of. The load Test planning and scoping assumes greater significance when testing with open source tools. A careful study of application architecture is also important to understand the protocols between client and first server along with the deployment model for monitoring. Limitations for open source tools have to be well understood and a workaround have to be found. For example, if user bandwidth simulation is a important criterion, third party bandwidth throttles can be integrated with open source tools to provide the same functionality.

As a conclusion, a careful evaluation of the open source tool to understand it's limitations and knowledge of available workarounds becomes very important for it's implementation success in any performance testing project.

Open Source Tools for Performance Testing:

Comparative Study

|Evaluation Criteria |Description |OpenSTA |Apache JMeter |WebLoad |

|Protocols |The communication |HTTP 1.0 / 1.1 / HTTPS (SSL),|HTTP,FTP, |HTTP/S, WAP, AJAX, ActiveX,|

| |protocols that can be |SOAP/XML |SOAP/XML-RPC,JDBC |Java, Web services. |

| |captured manipulated and| | |Multi-protocol scripts also|

| |replayed by the tool. | | |possible |

|Playback functions |Replaying of the script |Extended logging supports |It’s a GUI tool. So there |Extended log view, which |

| |and script debugging |view of parameter values and |are many GUI Listeners, |displays request and |

| |facilities. |Server messages. |which are used to capture |response data. |

| | | |the record and replay time| |

| | | |messages. | |

|Recording Protocol |The capability of tool |Available for HTTP\HTTPS |Available for the |Available for the supported|

|Data |to capture the protocol | |supported protocols |protocols |

| |specific data being | | | |

| |passed between client | | | |

| |and First Server. | | | |

|Scripting language |The medium used to |Called SCL, it uses a |Scripting is done using |WebLoad Test scripts are |

| |represent the captured |proprietary, “BASIC” like |the Beanshell/Java |created in Java Script, a |

| |protocol data and |language that has special |functions. However most of|familiar industry standard,|

| |manipulate the data for |automation scripting |the performance testing |eliminating the need to |

| |play-back. |facilities. Is limited in |features like Cookie |learn another proprietary |

| | |available functions, such as |Manager, Logic Controller,|language. |

| | |string manipulation and |Samplers etc are available| |

| | |supports direct DOM |using console. | |

| | |addressing. | | |

|Extensibility |The ability to increase |SCL script modules can be |The Beanshell/JAVA |In keeping with its open |

| |the functionality of the|defined in â??Includeâ?? |functions can be defined |standards approach, WebLOAD|

| |tool. |files. Open Source, therefore|and can be used as Plug-in|enables testers to |

| | |new tool functionality can be|as per the specific |seamlessly embed their own |

| | |added using C++. |requirements. |Java, ActiveX or COM |

| | | | |objects in the test script.|

| | | | |The flexible framework of |

| | | | |WebLOAD easily enables its |

| | | | |expansion in support of |

| | | | |other protocols, using the |

| | | | |WebLOAD Extensibility SDK. |

|Scripting Interface |The interfaces supplied |Has low-level HTTP protocol |They are many consoles |An intuitive interface |

| |by the tool application |view and provides graphical |provided by Meter which |enables novice testers to |

| |for the purpose of |tree representation of the |are used to edit the |create scripts with |

| |script editing. |Document Object Model (DOM) |script. The Beanshell/JAVA|drag-and-drop simplicity. |

| | |structure. Viewable captured |functions can also be used|Experienced users will |

| | |HTML rendering and |to enhance scripts. |appreciate the ability to |

| | |addressable server-header | |edit the script directly in|

| | |table. Some language | |the familiar java script |

| | |sensitive, syntax color | |language. |

| | |coding functionality | | |

| | | | | |

| | | | | |

| | | | | |

|Correlation |The task of substituting|Manual correlation using |Manual correlation using |SmartCopy feature of |

| |values in dynamic data |graphical interactive DOM |the console known as |WebLoad allows easy to |

| |to enable successful |structure. Facility to |Regular Expression |implement manual |

| |playback. |automatically generate script|Extractor is use to |correlation through GUI |

| | |code to aid variable |capture the value in the |feature without actually |

| | |substitution. |variable. |needing to do scripting for|

| | | | |the same. However this |

| | | | |feature is not available in|

| | | | |Open Source Version of |

| | | | |WebLoad |

| | | | | |

| | | | | |

| | | | | |

|Cookie Management |Detection, recording and|HTTP header cookies are |HTTP header cookies are |Automatic Sever Cookie |

| |playback of HTTP |managed automatically and can|managed automatically |management feature in |

| |cookies. Both tools need|be manipulated manually if |using the console known as|WebLoad allows user to |

| |additional code to |required. |HTTP Cookie Manager. |isolate himself from |

| |manage JavaScript | | |managing the dynamic serve |

| |generated cookies | | |side cookie. |

| | | | |Depending on the “Script |

| | | | |content” setting in the |

| | | | |“Record Options” dialog, |

| | | | |WebLOAD will record the |

| | | | |different headers into the |

| | | | |agenda, including the |

| | | | |cookie header. |

|Parameterization |Automatically changing |Extensive facilities for data|Parameterization can be |Facilities for data entry |

| |dynamic data values |entry including wizard |done in the console known |in the script through the |

| |passed from client to |interface to automatically |as Users Parameters. |Global data Entry file. The|

| |the server during POST |generate test data. Standard | |data file can be shared |

| |requests for more |functions for sequential, | |across virtual users |

| |accurate emulation of |random and pseudo-random | |running on the same machine|

| |real users. Often |data-file access. Have | |even under multiple |

| |essential for session |standard common locking | |iterations |

| |management. |facilities for maintaining | | |

| | |uniqueness of parameters for | | |

| | |an individual load injector | | |

| | |or across all injectors on a | | |

| | |distributed test. | | |

|Monitoring |Resource usage |NT and SNMP Monitors are |There are many consoles |Resource Monitoring is |

| |information is captured |available for monitoring the |which are known as |supported by running |

| |during execution. It can|hardware related resource |Listeners. These consoles |performance measurements |

| |be shown during |such as memory, CPU |give the performance of |manager (PMM) from behind |

| |execution and used to |utilization etc. |the servers. |the firewall fo the |

| |build performance | |Very basic monitoring for |supported servers. (Windows|

| |reports. | |Tomcat servers |NT/2000/XP |

| | | |• Monitors threads and |Unix platforms using SNMP |

| | | |memory usage |Unix platforms using RSTATD|

| | | |• More useful for stress | |

| | | |testing than monitoring |Windows .NET Framework |

| | | | |Microsoft IIS |

| | | | |Sun One (former iplanet 6) |

| | | | |Apache using SNMP) |

|Distributed tests |The ability to |Supports multiple |Supports multiple |Not Possible in open source|

| |distribute the |load-injectors managed by a |load-injectors managed by |version. The load can only |

| |generation of load |single controller. Uses |a single console |be given from one box in |

| |across multiple |TCP/IP if on the same network| |open source version of |

| |load-injector machines. |or the Web-relay feature uses| |Webload |

| | |HTTP to control | | |

| | |load-injectors located within| | |

| | |remote DMZs. | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

| | | | | |

|IP Spoofing |The ability to emulate |Not available. The requests |Not available out of box |Extended Multi IP support, |

| |the behavior of |going from one machine have |in JMeter. However the |which is of two types. |

| |different IP addresses |the same IP imprint |same can be emulated by |Per Client - preserving |

| |accessing a system. | |writing custom functions |current behavior, meaning |

| |Particularly useful with| |in Beanshell |different IP per client but|

| |Load balancing systems. | | |using the same IP for all |

| | | | |rounds. This is the default|

| | | | |value. |

| | | | |Per Round - support for |

| | | | |using a different IP from |

| | | | |the pool per client, per |

| | | | |round, until the pool is |

| | | | |exhausted then cycle back |

| | | | |to start. |

|Caching |The ability to emulate |Can control browser cache |Cache related functions |Not available in webload |

| |the caching of pages as |emulation during playback and|are available in |OpenSource |

| |performed by a web |control setting for each |Beanshell/JAVA which can | |

| |browser. |individual vuser. |be use to emulate the | |

| | | |caching of pages as | |

| | | |performed by a web. | |

|User connection speed |The ability to emulate |No inbuilt features to |It can be done with the |Not available in webload |

|emulation |the different network |emulate user-connection speed|help of some the functions|OpenSource |

| |speeds that can be used |emulation. |used in the Beanshell/JAVA| |

| |by real users. | |but it is not easy and not| |

| | | |yet used. | |

|Reporting & analysis |The facilities to |Simple charts and graphs |It creates some of the |WebLOAD Console displays |

| |examine and investigate |sufficient for analyzing key |graphs but reporting is |online reports of load |

| |the results of a test |load related statistics and |not available. |session statistics. The |

| |including timers and |resource usage monitors. | |user may create report |

| |monitored resources and |Resource usage monitors | |views that display any |

| |display the results in |supports graph overlays. Can | |subset of the wide array of|

| |graphical format. |be exported to Microsoft | |statistics collected by |

| | |Excel. No license | |WebLOAD. The user can |

| | |restrictions on OpenSTA | |toggle between a graphical |

| | |distribution thus stats can | |or textual (table) view of |

| | |be viewed by any user with | |each report. |

| | |access. -Free tools and excel| | |

| | |macros available through | |WebLOAD enables the user to|

| | |public user-forums. | |load results of a |

| | | | |previously executed session|

| | | | |for comparison with the |

| | | | |results of the current |

| | | | |session |

|Scalability |The ability of the tool |Main resource used is RAM. |Resource limitations are |Resource limitations are |

| |to generate numbers of |Tested for simple ASP pages, |number of threads and RAM.|number of threads and RAM. |

| |virtual users and the |reaching up to 3,000 users on|Approx 1 vuser per ½ Mb |On a machine with processor|

| |corresponding resource |load generator of 1Gb RAM on |RAM for NT/W2K. Windows |speed of 600/7333 MHZ and |

| |usage. Actual resource |a single P4 processor & W2K. |95, 98 & Unix are less |RAM of size 256MB, 100-150 |

| |use depends on the |Unconfirmed report of a limit|efficient. Max. Approx |threads could be run |

| |number, size and |of 1664 VUs per Win2K machine|1,500 VUs per PC. |successfully |

| |complexity of the |for complex scripts. | | |

| |scripts. |Suspected thread limitation. | | |

| | |No licensing limitations. | | |

References

1.

2. jakartha.

3.

4.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download