Archive for the ‘SLAs’ Tag

Performance Testing in Agile Framework

This is again not a new thing in the industry. It has been there for some time now. However, recent initiatives within the organization where I work got me thinking. At the first look performance testing within Agile framework do not seem to fit in as performance testing for a large part has always been done within the functionally-stable-code paradigm.

On the other side performance testing at an early stage looks to be a very good idea. The reason is there will be limited possibility of presence hotspots in code-base. The main performance bottlenecks that may be present, would be more on the infrastructure side which are more easier to fix and with almost no functional regression-testing.  Thus we do have a very good case for performance testing to be in Agile framework.

Although agile after each sprint delivers a workable product, there are some challenges that would crop-up here –

  1. Stakeholders Buy-in
  2. Project Management Vision
  3. Definition of the SLAs
  4. Unstable builds
  5. Development and Execution of Test Cases
  6. Highly skilled performance team

1. Stakeholders Buy-in

This is the most important aspect for any project. From a performance point of view this becomes altogether more important. Having an additional performance test team from the early phases of this project does put an additional strain on the budget and the timeline of the projects. Furthermore the advantages of performance testing are more of an intangible kind than say a functional testing. Thus it becomes a tad more difficult to get a stakeholders buy-in.

2. Project Management Vision

Despite the importance of the performance of an application, many project managers are still unaware of the performance test processes and the challenges a performance team faces. This apathy has been the undoing of many good projects. Within Agile framework, this would cause more conflicts between the project managers and performance teams. The ideal way would be to have a detailed meeting between the developers, performance architect/team and project managers and chalk out a plan which would decide the following –

  • Modules to be targeted for performance testing
  • Approach of performance testing
  • In which sprint to begin of performance testing?
  • Identification of Module-wise load that needs to be simulated
  • Profiling Tool Selection
  • Transition of Performance Tool – This would happen, as initially tools like JPerfUnit may be used and at a later stage end-to-end load testing tools like LoadRunner or SilkPerformer may be used.

These are some of the many parameters that needs to be defined at a project management level.

3. Definition of SLAs

This will be one of the most challenging aspect as business can always give an insight on end-to-end SLA. However, at a module-level it becomes more challenging. Here there arises the need for developer and performance architect to arrive at an estimated number. Apart from this,  there may be a need for de-scoping of some modules as not all modules may be critical from performance stand-point.

4. Development and Execution of Test Cases

Although agile delivers a workable product at the end of every sprint, we may not be able to use the standard load testing tools as not all components would be present which a standard load-testing tool may require.  So in most cases, there is an inherent challenge in simulating the load.  Tools may very between phases. For instance, JPerfUnit, a  stub or test harness might be used in the initial phases.  An extra development time for this would have to be estimated in the initial project planning phase. Finally, once created, development and execution of test cases would follow. With the help of profiling tool, most performance hotspots can be identified early.

At later stages of the project, the traditional performance scripting and execution will replace the harness that was created in the earlier stage. So there will be a big re-work which again would be a challenge considering the sprint-timeline. So sprint should be planned taking this into consideration too.

5. Unstable builds

As development and testing is going on simultaneously, a break-off time would be needed within a sprint. The other alternative is to build the script on a continuous basis based which is quite difficult. Performance team would then create the required test script and test the code. This break-off point is only to enable performance team to work on a particular test case. If major changes occur after the break-off time, they will have to be incorporated in the next sprint.

6. High-skilled Performance Team

Last but not the least as the cliché goes. More than the skill-level, it is imperative that team members should have the confidence to get their hands dirty learning about the system and innovating as challenges continue to crop-up during the various sprints, keeping in mind the performance test approach, and thus ensuring a quality product.

The agile methodology does present lot of challenges at least from performance stand-point. However, if the performance angle is kept in mind from the beginning, it will certainly help in reducing lot of pain later. From a performance tester stand-point, agile is a gold-mine, as you get involved with developers at an early stage of the application. This would help in greater understanding of the application and its internal workings which eventually would help in better identification of performance bottlenecks.


So those are my ideas… Looking forward for your thoughts on this..


Why do performance testers need to know business?


Well the question does seem easy enough at the first look. This is what the managers across the board have been crying hoarse. Testers need to know business. Or else how can they test? Agreed. But when it comes to performance testers, does this hold good?

From a performance testing standpoint, what does a normal performance testers do? In almost all cases, the only way where the critical business flows are identified is via the hits on the web server. If it is a new application, most of the time, the business will have a fair idea of which flows will be critical and those will be handed down to the performance testers.

Now our performance tester will come in and write scripts, designs the scenario based on the requirements and executes the test. During the analysis, the following things will be looked at.

1. Server Health
2. Performance issues in code
3. SLAs

So keeping this in mind where does the business come in? Does it really matter to the performance tester to understand the business? The only that matters to him is whether the SLA has been achieved or if there are some hidden performance bugs which may crop up apart from the server health. Everything related to do with the technology and almost nothing with business except the SLAs, if there are any. Experienced performance testers, do not even need to see the flow and they can make the script robust. On the other hand a functional tester cannot do without knowing the business, with the automation engineer falling somewhere in between these two extremes. Is this the correct way of looking at it?

Now coming to the answer to the question in the title of the post. Every business for its survival have to create a positive impression on its customers/clients. Any customer who interacts via a business transaction takes with him/her an experience.  And this experience is what makes the customer comes back again and again building a relationship with the business. Out in a retail shop, businesses control this by having good sales force, good ambience, etc. However, online the only experience that a user can get is the look and feel of the application, ease of traversing and the speed with which the customer’s job is done. Thus performance testers directly impact the bottom-line of the business as they are responsible for this user experience. Thus it becomes imperative for performance testers to know the business, from where the revenue comes to ensure the application creates a good user experience which in turn helps the business to grow!

PS: The other testing is also equally important. However, I just wanted to bring out the importance of a business knowledge for performance testers and their impact on business which is much more than what most performance testers tend to believe.

%d bloggers like this: