Here is a quick summary of our catch up with Adam:
How has the ecosystem for performance/load testing changed last few years and what trends are in store for future?
There are a lot of commercial and open source tools as well as on line services available to performance testers now. This is largely driven by the historically cost prohibitive nature of load testing software since the launch of LoadRunner in 1994. Despite best efforts and significant investment no company has yet been able to achieve the same rate of penetration as that that was seen with LoadRunner. Even with the Microfocus Acquisition of both SilkPerformer from Segue and QALoad from Compuware there has still been no change to the dominance of the market leader.
|Despite best efforts and significant investment no company has yet been able to achieve the same rate of penetration as that that was seen with LoadRunner.|
This is because these companies offer a similar product at a reduced cost, but in the end the customer / user is not interested in saving just a few bucks for the hassle and stress and risk related to a very large change to their provider.
Open source has made an impact but as load testers are already relatively rare, the load testers capable of using these open source technologies are even rarer due to the technical nature of the tools, lack of training availability and lack of support. This reduces interest by customers as they do not want to invest time and effort into something that might be difficult to support in the future. We can take the example of OpenSTA : considering that last update was in 2007, the user base can only decline.
We are seeing that SaaS delivery is changing the game as there is no re-training or deployment costs and it’s an easy to swallow proposition. However it’s not suitable for all, for example internal applications inside the firewall, suddenly the proposition cannot work.
As with the IOS / Android style apps market we see that ‘freemium’ software bridges this gap – it needs to be sophisticated enough to deal with the modern complex applications where more and more logic resides in the browser, and it needs to be free to give the end users the chance to pick it up and learn it, run small projects and get immediate benefit from it.
It’s a win-win situation: once users educate themselves using free video tutorials on the web site they can save their employers (or in the case of contractors or consultancies their customers) a lot of time and money in return for their expertise. In fact a small company could easily now become a cloud service provider utilizing our software AgileLoad running on Amazon AWS ! (See our cloud videos if you’re interested)
The current trends in software development are around rich client technologies, Agile processes, cloud platforms, mobile usage, growing expectation on performance and cost optimization. AgileLoad is following these trends and is developed to give flexibility to project team.
AgileLoad can be installed on a private lab, a cloud infrastructure or a mix of both with no restrictions or extra cost. The product is free to download and free for scripting. We only charge by test runs from 50 VU on a daily basis. This means that the tester can build and setup his test (scripts, profiles, reports) as he want, he just pay the effective execution.
What are 3 common mistakes that enterprises do defining and implementing performance test strategy as a whole and selecting a solution?
- Underestimating the complexity of generating realistic load against a complex system : Essentially this is a reverse engineering process of the client side logic (which these days can be very complex as we have seen with Ajax and Flash and now HTML5). We have addressed this in our product with technology that can learn how to deal with complex web transactions.
- Over specifying load and not relating it to real use scenarios: this can be a waste of time and money – for example fixing a bottleneck that will never come into existence during overload conditions
- Talking about virtual users exclusively rather than a mix of transaction rates, response times, virtual users and SLA’s or performance expectations results
Please provide insights into AgileLoad and how different or competitive is it compared to others?
Five great reasons to use AgileLoad on your next performance testing project:
||AgileLoad can learn how to manage complex transactions using Automatic Modeling.|
- Cost optimization – Build all your load test scenarios and setup your tests without paying any fee. You just pay for the test execution.
- Full support of dynamic web and mobile technologies
- Vastly Reduced Scripting Time – AgileLoad can learn how to manage complex transactions using Automatic Modeling. When this was demonstrated at the STPCon OnLine summit earlier this year it was one of the highlights of all the technology seen during that event. Using Automatic modeling the 1st script is created and during the parameterization process (tying up of session ID’s and other essential parameters) it is possible to teach AgileLoad how to manage these parameters which saves time not only in the 1st script but in all subsequent scripts against the same application
- High Performance System Quickly – Agileload is featured with pre-set performance indicators and anomaly profiles currently used by experts. You can diagnose and analyze bottlenecks without technology expertise. Using built in knowledge, AgileLoad combs through the monitoring results (database, web server, operating systems etc.) and list detected anomalies. Remediation advice is then given to enable tuning so that the system is ready for retest to find any additional bottlenecks.
- Executive Summary of the performance profile of your application – Automatic report generation means that reports are on your desktop within less than a minute of the test completing. You customize your own reports templates and reuse them when needed. These reports can be tailored for different stakeholders such as DBA, ecommerce manager etc. with each report containing relevant data for each stakeholder.
ROI and performance testing has been a subject of discussion for years but there seems to be a genuine lack of understanding of this subject at a wider scale when it comes to implementation. Your thoughts on how to define and measure it at an enterprise level?
Lets say that the cost of poor application performance is only evident when a problem arise. As a consequence, the overhead of tools and staff to implement a performance strategy may not be obvious and can be seen as an insurance policy.
|But all that means still to put “performance” as a company priority and it doesn’t insure that application will not effectively crash during the next marketing event. The only way to be sure of it is actually to test it !|
It depends also a lot on the size and complexity of companies and projects. In some case performance testing in-house makes the most sense economically, while for other outsourcing rules better.
Thus, to define which strategy is better at enterprise level you need to build your own test cases:
- How much does it cost to the company? Which tools better fits the current and future applications needs, their licenses and training prices? What extra infrastructure is necessary? What would be the average number of hours needed per project and per labor cost ?
- On the other ends, what are the potential incomes gains of a performance optimized application? There is a lot of statistics published about it. It is demonstrated that a performance optimized application improves user experience and employee productivity. It also increases traffic and sales while reducing user support.
- So an analysis of current analytics data about conversion rate, transaction time, number of visits, and a projection on how it could be improve gives you consequently the extra income gains.
This analysis helps to have a better idea of the ROI specific to an organization. One can argue that Performance testing is not the only approach to alleviate performance related risks at enterprise level. We can also think of:
- Implementing performance monitoring tools in production to get a better understanding of what’s happening
- Use of performance best practices through development
- Use of a scalable infrastructure
But all that means still to put “performance” as a company priority and it doesn’t insure that application will not effectively crash during the next marketing event. The only way to be sure of it is actually to test it !
Is there a way to load or performance test without using a tool? ………
Members of the AgileLoad team have seen this done using a pizza approach – lay on free Pizza and maybe even beer in return for after hour time from staff.
There are issues with this approach as follows:
- Not repeatable – or at least not readily repeatable.
- Absolutely not possible for very large tests (thousands of users) for all but the largest of organizations – in reality only the IT team can be asked to run such an exercise.
- Not accurate – take an automation like AgileLoad – it runs scripts time after time using different data for each iteration. Ask a real user to that and they will quickly be bored and will start to make mistakes as they are asked to repeat tasks that one real life user would do once. There are timing issues – how do you record time accurately – stop watch and graph paper, what about when to start each task and when to stop? Also to really simulate unique users each of those users must clear their cache and history for each transaction – there is a technical ‘can of worms’ taking this approach.
So yes while it’s possible it’s not really an effective way to build quality into an application – it might just scrape through as an experimental evaluation of the performance of the app but that’s all.