Archive for the ‘business case’ Tag

Performance Center: A ground up view

To begin with I was a big fan of Performance Center. I always saw it as tool built on the lines of Quality Center and thought it great for businesses to go for a performance center. However when I first started using it doubts started creeping in. And my doubts were confirmed once I got to use it extensively. So given here are my observations:

  • To start with Performance Center is simply a wrapper over LoadRunner. Whatever, Performance Center (PC) does from the browser, set of commands are fired to the machines that has controller to start the testing. The benefits are given below:
    • No need for the user to log-in to a controller machine
    • No need for the user to log-in to load generators.
  • These are all true. But the catch is, to begin a new test, PC takes almost 5-15 min to connect to Controller and Load generator. So if a test is stopped and started again average of 20 mins of time is lost in connection to the servers. On the other hand, same task happens instantaneously on Controller.
  • Additional infrastructure is needed to deploy the performance center on server class machines. If the performance center is not fast enough all the testing will be impacted because of this.
  • HP LoadRunner always had the issue of controller crashing during a test. With the introduction of PC, complexity has increased and crashes have become more frequent.
  • Additional man-power is also needed to manage performance center. Any organization which deploys PC will need PC administrators to address all issues arising out of PC as well as user management.
  • The main benefit that HP talks about in implementing PC is the centralized management of resources and creation of Performance CoE. Both benefits can be achieved without PC. A Performance CoE is not just standardization of processes. It also has other features such as knowledge management, research and development, technical competencies of the resources of CoE, metrics etc. PC is simply a cog in the complete machinery of CoE.
  • The other benefit is the version control. Till PC 9.5, there was no version controlling in the scripts. But a work-around was there by keeping the script in separate project. However, for the scenario design version-control is zilch. A better version-control can be implemented at free-of-cost by using open source Subversion. Both scripts and scenarios can  be version-controlled using Subversion and it gives a assurance on the scenario that gets executed. We simply have to create a new scenario from scratch in PC.
  • Direct integration with QC. Yet this is not as important feature as it is made out to be. The principle reason is unlike QA (manual / automation) defects, pure performance defects are generally very less. So even if there is no direct integration   with QC, defects can be easily tracked through QC.
  • If there are specific settings that needs to be in load generators such as setting up the classpath for JAVA scripts, it still needs to be done manually by logging into the load generators. This follows my first point. PC is just a wrapper over LoadRunner.
  • The other benefits of finding expensive performance bugs, improving response times are the benefits that LoadRunner already have.
  • Implementation of standard processes to bring in efficiency is also touted as an additional benefit. However, performance testing starts with asking the right questions for requirement, architecture analysis, creation of test plans, monitoring strategy, etc. Performance Center can only help in listing out the requirements and execution of test. The rest loadrunner does it with or without performance center. Requirement gathering and test result analysis is still the tester’s job.
  • Moreover, PC 11 claims to generate good reports. Unlike QC, where defect reports gives, good insights into overall testing status, performance report need quite a bit of human intervention. You really need an expert to provide meaning to the numbers and a proper conclusion which a tool can not achieve at this stage.
  • Additionally LR Analysis is a great tool, and it works admirably well for collating and compiling the duration test results. PC simply fails to  create analysis reports when the data is huge. Moreover downloading huge raw results, itself becomes a bottleneck during the analysis phase.
  • Scripting still remains local and PC can do little in that.
  • One advantage that PC has over LR is that all the stakeholders can view what is happening while the test is running. From a tester’s standpoint that is duplicity of efforts. Performance tester will be watching over the test plus the server health statistics to uncover potential performance bottlenecks. Management can at most say test is good or bad but still will have to wait till performance tester gives a thorough analysis of the test.
  • Second advantage that PC can  provide is when managing huge number of projects through a limited number of controllers and load-generators. However, this can be easily achieved using the Outlook calendar or even common excel that is shared. It doesn’t make a business sense to buy something expensive that can be achieved easily through an existing tool.
  • Finally LoadRunner is a desktop application while PC is web application and web application can never be as fast as desktop applications. This is one huge disadvantage which PC cannot overcome.

So although on paper and in presentations, PC does look to be a great tool for top management, but when looked from ground-zero, it doesn’t match up to the expectations and a great many dollars can be saved by avoiding going for PC and sticking to LR itself. The point is if you already have paid for LR, there is no real business reason to go for PC.

So welcoming your thoughts and leaving with the following thought:(Image Courtesy)

Disclaimer: Author has worked till PC 9.5. Regarding PC 11.0 information has been collected from HP’s website.

Advertisements
%d bloggers like this: