Showing posts with label LoadRunner Basics. Show all posts
Showing posts with label LoadRunner Basics. Show all posts

May 12, 2016

LoadRunner Part # 7 - Performance Testing Roadmap

Performance Testing Roadmap: Detailed Steps

Performance Testing Roadmap can be broadly divided into 5 steps:
  • Planning for Load Test
  • Create VUGen Scripts
  • Scenario Creation
  • Scenario Execution
  • Results Analysis (followed by system tweaking)
Now that you've LoadRunner installed, let's understand the steps involved in the process one by one.




Steps involved in Performance Testing process

Planning for the Load Test

Planning for Performance Testing is different from planning an SIT (System Integration Testing) or UAT (User Acceptance Testing). Planning can be further divided into small stages as described below:

Assemble Your Team

When getting started with Performance Testing, it is best to document who will be participate in the activity from each team involved during the process.
Project Manager:
Nominate the project manager who will own this activity and serve as point person for escalation.
Function Expert/ Business Analyst:
Provide Usage Analysis of SUL & provides expertise on business functionality of website/SUL
Performance Testing Expert:
Creates the automated performance tests and executes load scenarios
System Architect:
Provides blueprint of the SUL
Web Developer and SME:
  • Maintains website & provide monitoring aspects
  • Develops website and fixes bugs
System Administrator:
  • Maintains involved servers throughout testing project

Outline applications and Business Processes involved:

Successful load testing requires that you plan to carry out certain business process. A Business Process consists of clearly defined steps in compliance to desired business transactions - so as to accomplish your load testing objectives.
A requirements metric can be prepared to elicit user load on the system. Below is an example from an attendance system in a company:
In the above example, the figures mention the number of users connected to the application (SUL) at given hour. We can extract the maximum number of users connected to a business process at any hour of the day which is calculated in the right most columns.
Similarly, we can conclude the total number of users connected to the application (SUL) at any hour of the day. This is calculated in the last row.
The above 2 facts combined give us the total number of users with which we need to test the system for performance.

Define Test Data Management Procedures

Statistics and observations drawn from Performance Testing are greatly influenced by numerous factors as briefed earlier. It is of critical significance to prepare Test Data for Performance Testing. Sometimes, a particular business process consumes a data set and produces a different data set. Take below example:
  • A user 'A' creates a financial contract and submits it for review.
  • Another user 'B' approves 200 contracts a day created by user 'A'
  • Another user 'C' pays  about 150 contracts a day approved by user 'B'
In this situation, User B need to have 200 contracts 'created' in the system. Besides, user C needs 150 contracts as "approved" in order to simulate load of 150 users.
This implicitly means, that you must create at least 200+150= 350 contracts.
After that, approve 150 contracts to serve as Test data for User C - the remaining 200 contracts will serve as Test Data for User B.
Outline Monitors
Speculate each and every factor which could possible affect the performance of system. For example, having reduced hardware will have potential impact on the SUL(System Under Load) performance.
Enlist all factors and setup monitors so you can gauge them. Here are few examples:
  • Processor    (for Web Server, Application Server, Database Server and Injectors)
  • RAM   (for Web Server, Application Server, Database Server and Injectors)
  • Web/App Server (for example IIS, JBoss, Jaguar Server, Tomcat etc)
  • DB Server (PGA and SGA size in case of Oracle and MSSQL Server, SPs etc.)
  • Network bandwidth utilization
  • Internal and External NIC in case of clustering
  • Load Balancer (and that it is distributing load evenly on all nodes of clusters)
  • Data flux   (calculate how much data moves to and from client and server - then calculate if capacity of NIC is sufficient to simulate X number of users)

Create VUGen Scripts

Next step after planning is to create VUser scripts.

Scenario Creation

Next step is to create your Load Scenario

Scenario Execution

Scenario execution is where you emulate user load on the server by instructing multiple VUsers to perform tasks simultaneously.
You can set the level of load by increasing and decreasing the number of VUsers that perform tasks at the same time.
This execution may result the server to go under stress and behave abnormally. This is the very purpose of the performance Testing. The results drawn are then used for detailed analysis and root cause identification.

Results Analysis (followed by system tweaking)

During scenario execution, LoadRunner records the performance of the application under different loads. The statistics drawn from test execution are saved and details analysis is performed. The 'HP Analysis' tool generates various graphs which help in identifying the root causes behind lag of system performance, as well as system failure.
Some of the graphs obtained include:
  • Time to First buffer
  • Transaction Response Time
  • Average Transaction Response Time
  • Hits Per Second
  • Windows Resources
  • Errors Statistics
  • Transaction Summary

FAQ

Which Applications should we Performance Test ?

Performance Testing is always done for client server based systems only. This means, any application which is not a client-server based architecture, must not require Performance Testing.
For example, Microsoft Calculator is neither client-server based nor it runs multiple users; hence it is not a candidate of Performance Testing.

What is the difference between Performance Testing & Performance Engineering

It is of significance to understand the difference between Performance Testing and Performance Engineering. An understanding is shared below:
Performance Testing is a discipline concerned with testing and reporting the current performance of a software application under various parameters.
Performance engineering is the process by which software is testedandtuned with the intent of realizing the required performance. This process aims to optimize the most important application performance trait i.e. user experience.
Historically, testing and tuning have been distinctly separate and often competing realms. In the last few years, however, several pockets of testers and developers have collaborated independently to create tuning teams. Because these teams have met with significant success, the concept of coupling performance testing with performance tuning has caught on, and now we call it performance engineering.

LoadRunner Part # 6 - Architecture

What is LoadRunner Architecture?

Broadly speaking, the architecture of LoadRunner is complex, yet easy to understand.
Suppose you are assigned to check performance of Amazon.com for 5000 users
   In a real life situation, these all these 5000 users will not be at homepage but in different section of the websites. How can we simulate different 
VUGen:
VUGen or Virtual User Generator is an IDE (Integrated Development Environment) or a rich coding editor. VUGen is used to replicate System Under Load (SUL) behaviour. VUGen provides a "recording" feature which records communication to and from client and Server in form of a coded script - also called VUser script.
So considering the above example, VUGen can record to simulate following business processes:
  1. Surfing the Products Page of Amazon.com
  2. Checkout
  3. Payment Processing
  4. Checking MyAccount Page
Controller
Once a VUser script is finalized, Controller is the main component which controls the Load simulation by managing, for example:
  • How many VUsers to simulate against each business process or VUser Group
  • Behaviour of VUsers (ramp up, ramp down, simultaneous or concurrent nature etc.)
  • Nature of Load scenario e.g. Real Life or Goal Oriented or verifying SLA
  • Which injectors to use, how many VUsers against each injector
  • Collate results periodically
  • IP Spoofing
  • Error reporting
  • Transaction reporting etc.
Taking analogy from our example controller will add following parameter to the VUGen Script
1) 3500 Users are Surfing the Products Page of Amazon.com
2) 750 Users are in Checkout
3) 500 Users are performing Payment Processing
4) 250 Users are  Checking MyAccount Page ONLY after 500 users have done Payment Processing
Even more complex scenarios are possible
  1. Initiate 5 VUsers every 2 seconds till a load of 3500 VUsers (surfing Amzon product page) is achieved.
  2. Iterate for 30 minutes
  3. Suspend iteration for 25 VUsers
  4. Re-start 20 VUSers
  5. Initiate  2 users (in Checkout, Payment Processing , MyAccounts Page) every second.
  6. 2500 VUsers will be generated at Machine A
  7. 2500 VUsers will be generated at Machine B
Agents Machine/Load Generators/Injectors
LoadRunner Controller is responsible to simulate thousands of VUsers - these VUsers consume hardware resources for example processor and memory - hence putting a limit on the machine which is simulating them. Besides, Controller simulates these VUsers from the same machine (where Controller resides) & hence the results may not be precise. To address this concern, all VUsers are spread across various machines, called Load Generators or Load Injectors.
As a general practice, Controller resides on a different machine and load is simulated from other machines. Depending upon the protocol of VUser scripts and machine specifications, a number of Load Injectors may be required for full simulation. For example, VUsers for an HTTP script will require 2-4MB per VUser for simulation, hence 4 machines with 4 GB RAM each will be required to simulate a load of 10,000 VUsers.
Taking Analogy from our Amazon Example, the output of this component will be
Analysis:
Once Load scenarios have been executed, the role of "Analysis" component comes in.
During the execution, Controller creates a dump of results in raw form & contains information like, which version of LoadRunner created this results dump and what were configurations.
All the errors and exceptions are logged in a Microsoft access database, named, output.mdb. The "Analysis" component reads this database file to perform various types of analysis and generates graphs.
These graphs show various trends to understand the reasoning behind errors and failure under load; thus help figuring whether optimization is required in SUL, Server (e.g. JBoss, Oracle) or infrastructure.
Below is an example where bandwidth could be creating bottleneck. Let's say Web server has 1GBps capacity whereas the data traffic exceeds this capacity causing subsequent users to suffer. To determine system caters to such needs, Performance Engineer needs to analyse application behaviour with abnormal load. Below is a graph LoadRunner generates to elicit bandwidth.

LoadRunner Part # 5 - Inroduction to Loadrunner


Guide to Download and Install HP LoadRunner 12.0 - here

Introduction to HP LoadRunner



LoadRunner is a Performance Testing tool which was pioneered by Mercury in 1999. Loadrunner was later acquired by HP in 2009.
LoadRunner supports various development tools, technologies and communication protocols. In fact this is the only tool in market which supports such large number of protocols to conduct performance testing.
LoadRunner is not only pioneer tool in Performance Testing, but it is still market leader in the Performance testing paradigm. In a recent assessment, LoadRunner has about 85% market share in Performance Testing industry. 
Broadly, LoadRunner supports RIA (Rich Internet Applications), Web 2.0 (HTTP/HTML, Ajax, Flex and Silverlight etc.), Mobile, SAP, Oracle, MS SQL Server, Citrix, RTE, Mail and above all, Windows Socket. There is no competitor tool in the market which could offer such wide variety of protocols vested in single tool.
What is more convincing to pick LoadRunner for performance testing is the credibility of this tool. LoadRunner has long established reputation as often you will find clients cross verifying your performance benchmarks using LoadRunner. You'll find relief if you're already using LoadRunner for your performance testing needs.
LoadRunner is tightly integrated with other HP Tools like Unified Functional Test (QTP) & ALM (Application Lifecycle Management) with empowers you to perform your end to end Testing Processes. 
LoadRunner works on a principal of simulating Virtual Users on the subject application. These Virtual Users, also termed as VUsers, replicate client's requests and expect corresponding response to pass a transaction.

Why do you need Performance Testing?

An estimated loss of 4.4 billion in revenue is recorded annually due to poor web performance.
In today's age of Web 2.0, users click away if a website doesn't respond within 8 seconds. Imagine yourself waiting for 5 seconds when searching over Google or making a friend request on Facebook. The repercussions of performance downtime are often more devastating than ever imagined. We've examples such as those that recently hit Bank of America Online Banking, Amazon Web Services, Intuit or Blackberry.
According to Dunn & Bradstreet, 59% of Fortune 500 companies experience an estimated 1.6 hours of downtime every week. Considering the average Fortune 500 company with a minimum of 10,000 employees is paying $56 per hour, the labour part of downtime costs for such an organization would be $896,000 weekly, translating into more than $46 million per year.
Only a 5 minute downtime of Google.com (19-Aug-13) is estimated to cost the search giant as much as $545,000.
Its estimate that companies lost sales worth $1100 per second due to a recent Amazon Web Service Outage.
When a software system is deployed by an organization, it may encounter many scenarios that possibly result in performance latency. A number of factors cause decelerating performance, few example may include:
  • Increased number of records present in the database
  • Increased number of simultaneous requests made to the system
  • larger number of users accessing the system at a time as compared to the past

May 11, 2016

LoadRunner Part # 4 - Performance Parameters

Performance Parameters Monitored

The basic parameters monitored during performance testing include:
performance testing parameters
  • Processor Usage - amount of time processor spends executing non-idle threads.
  • Memory use - amount of physical memory available to processes on a computer.
  • Disk time - amount of time disk is busy executing a read or write request.
  • Bandwidth - shows the bits per second used by a network interface.
  • Private bytes - number of bytes a process has allocated that can't be shared amongst other processes. These are used to measure memory leaks and usage.
  • Committed memory - amount of virtual memory used.
  • Memory pages/second - number of pages written to or read from the disk in order to resolve hard page faults. Hard page faults are when code not from the current working set is called up from elsewhere and retrieved from a disk.
  • Page faults/second - the overall rate in which fault pages are processed by the processor. This again occurs when a process requires code from outside its working set.
  • CPU interrupts per second - is the avg. number of hardware interrupts a processor is receiving and processing each second.
  • Disk queue length - is the avg. no. of read and write requests queued for the selected disk during a sample interval.
  • Network output queue length - length of the output packet queue in packets. Anything more than two means a delay and bottlenecking needs to be stopped.
  • Network bytes total per second - rate which bytes are sent and received on the interface including framing characters.
  • Response time - time from when a user enters a request until the first character of the response is received.
  • Throughput - rate a computer or network receives requests per second.
  • Amount of connection pooling - the number of user requests that are met by pooled connections. The more requests met by connections in the pool, the better the performance will be.
  • Maximum active sessions - the maximum number of sessions that can be active at once.
  • Hit ratios - This has to do with the number of SQL statements that are handled by cached data instead of expensive I/O operations. This is a good place to start for solving bottlenecking issues.
  • Hits per second - the no. of hits on a web server during each second of a load test.
  • Rollback segment - the amount of data that can rollback at any point in time.
  • Database locks - locking of tables and databases needs to be monitored and carefully tuned.
  • Top waits - are monitored to determine what wait times can be cut down when dealing with the how fast data is retrieved from memory
  • Thread counts - An applications health can be measured by the no. of threads that are running and currently active.
  • Garbage collection - has to do with returning unused memory back to the system. Garbage collection needs to be monitored for efficiency.

Performance Test Tools

There are a wide variety of performance testing tools available in market. The tool you choose for testing will depend on many factors such as types of protocol supported , license cost , hardware requirements , platform support etc. Below is a list of popularly used testing tools.
  • HP Loadrunner - is the most popular performance testing tools on the market today. This tool is capable of simulating hundreds of thousands of users, putting applications under real life loads to determine their behavior under expected loads. Loadrunner features a virtual user generator which simulates the actions of live human users.
  • HTTP Load - a throughput testing tool aimed at testing web servers by running several http or https fetches simultaneously to determine how a server handles the workload.
  • Proxy Sniffer - one of the leading tools used for load testing of web and application servers. It is a cloud based tool that's capable of simulating thousands of users. Summary

LoadRunner Part # 3 - Performance Testing Process

Performance Testing Process

The methodology adopted for performance testing can vary widely but the objective for performance tests remain the same. It can help demonstrate that your software system meets certain pre-defined performance criteria. Or it can help compare performance of two software systems. It can also help identify parts of your software system which degrade its performance.
Below is a generic performance testing process
performance_testing_process
  1. Identify your testing environment - Know your physical test environment, production environment and what testing tools are available. Understand details of the hardware, software and network configurations used during testing before you begin the testing process. It will help testers create more efficient tests.  It will also help identify possible challenges that testers may encounter during the performance testing procedures.
  2. Identify the performance acceptance criteria - This includes goals and constraints for throughput, response times and resource allocation.  It is also necessary to identify project success criteria outside of these goals and constraints. Testers should be empowered to set performance criteria and goals because often the project specifications will not include a wide enough variety of performance benchmarks. Sometimes there may be none at all. When possible finding a similar application to compare to is a good way to set performance goals.
  3. Plan & design performance tests - Determine how usage is likely to vary amongst end users and identify key scenarios to test for all possible use cases. It is necessary to simulate a variety of end users, plan performance test data and outline what metrics will be gathered.
  4. Configuring the test environment - Prepare the testing environment before execution. Also, arrange tools and other resources.
  5. Implement test design - Create the performance tests according to your test design.
  6. Run the tests - Execute and monitor the tests.
  7. Analyze, tune and retest - Consolidate, analyze and share test results. Then fine tune and test again to see if there is an improvement or decrease in performance. Since improvements generally grow smaller with each retest, stop when bottlenecking is caused by the CPU. Then you may have the consider option of increasing CPU power.

LoadRunner Part # 2 - Types of performance testing

Types of performance testing


  • Load testing - checks the application's ability to perform under anticipated user loads. The objective is to identify performance bottlenecks before the software application goes live.
  • Stress testing - involves testing an application under extreme workloads to see how it handles high traffic or data processing .The objective is to identify breaking point of an application.
  • Endurance testing - is done to make sure the software can handle the expected load over a long period of time.
  • Spike testing - tests the software's reaction to sudden large spikes in the load generated by users.
  • Volume testing - Under Volume Testing large no. of. Data is populated in database and the overall software system's behavior is monitored. The objective is to check software application's performance under varying database volumes.
  • Scalability testing - The objective of scalability testing is to determine the software application's effectiveness in "scaling up" to support an increase in user load. It helps plan capacity addition to your software system.

Common Performance Problems

Most performance problems revolve around speed, response time, load time and poor scalability. Speed is often one of the most important attributes of an application. A slow running application will lose potential users. Performance testing is done to make sure an app runs fast enough to keep a user's attention and interest. Take a look at the following list of common performance problems and notice how speed is a common factor in many of them:
  • Long Load time - Load time is normally the initial time it takes an application to start. This should generally be kept to a minimum. While some applications are impossible to make load in under a minute, Load time should be kept under a few seconds if possible.
  • Poor response time - Response time is the time it takes from when a user inputs data into the application until the application outputs a response to that input. Generally this should be very quick. Again if a user has to wait too long, they lose interest.
  • Poor scalability - A software product suffers from poor scalability when it cannot handle the expected number of users or when it does not accommodate a wide enough range of users. Load testing should be done to be certain the application can handle the anticipated number of users.
  • Bottlenecking - Bottlenecks are obstructions in system which degrade overall system performance. Bottlenecking is when either coding errors or hardware issues cause a decrease of throughput under certain loads. Bottlenecking is often caused by one faulty section of code. The key to fixing a bottlenecking issue is to find the section of code that is causing the slow down and try to fix it there. Bottle necking is generally fixed by either fixing poor running processes or adding additional Hardware. Some common performance bottlenecks are
    • CPU utilization
    • Memory utilization
    • Network utilization
    • Operating System limitations
    • Disk usage

LoadRunner Part # 1 - Performance testing

Performance testing 


What is performance testing?

Software performance testing is a means of quality assurance (QA). It involves testing software applications to ensure they will perform well under their expected workload.
Features and Functionality supported by a software system is not the only concern. A software application's performance like its response time, do matter. The goal of performance testing is not to find bugs but to eliminate performance bottlenecks

The focus of Performance testing is checking a software program's
  • Speed - Determines whether the application responds quickly
  • Scalability - Determines maximum user load the software application can handle.
  • Stability - Determines if the application is stable under varying loads


Why do performance testing?

why_performance_testing
Performance testing is done to provide stakeholders with information about their application regarding speed, stability and scalability. More importantly, performance testing uncovers what needs to be improved before the product goes to market. Without performance testing, software is likely to suffer from issues such as: running slow while several users use it simultaneously, inconsistencies across different operating systems and poor usability. Performance testing will determine whether or not their software meets speed, scalability and stability requirements under expected workloads. Applications sent to market with poor performance metrics due to non existent or poor performance testing are likely to gain a bad reputation and fail to meet expected sales goals.Also, mission critical applications like space launch programs or life saving medical equipments should be performance tested to ensure that they run for a long period of time without deviations.