I have to say: “I am a big fan of automation”, so I am happy to see that more and more projects integrate continuous integration solutions to check if commits are correct and unit tests are running. In my on-going mission to automate everything (that is meaningful to automate) I usually don’t stop at continuous integration, but want to automate the release and deploy process as well, continuous delivery here we come.
But as I am also a big fan of application performance, automation does not stop here either. Just think about it, you already have everything automated: your application is working and already deployed and running on your testing environment. You already have testing for functional correctness, but what about non-functional correctness. The road is clear for loadtests as well. Perhaps not the full-fledged “let’s see what this system can do” loadtest (a.k.a. stresstest), but why not execute a meaningful loadtest automatically.
In this post I will show how one can easily integrate jmeter (http://jmeter.apache.org) loadtests with Jenkins.
Step 1: Automate loadtest tool and scripts deployment
In order to automate your loadtest you should also automate the deploy of the loadtesting tool and all necessary files to run the loadtest (scripts, datapools, …). Just a few pointers: The loadtest script and data belongs to the project like the junit tests do as well, so keep them in the configuration management with the project. If you use jmeter as loadtesting tool you can easily just download it on-demand if it is not yet available on the test environment. If you can get your hands on a server in your organisation where you can put the file, even better. At NovaTec we use a simple FTP server.
Step 2: Automate the execution of your JMeter loadtest
As you are automating the execution of your loadtest, you want to start jMeter in non-gui mode. You can easily do that in multiple ways. The easiest way is simply to execute the jmeter startscript
jmeter -n -t [path_to_jmx_file] -l [path_to_result_file]
To automate this execution you can simply integrate this call in your favorite build tool. I love Ant, so here is an example of how you can integrate that in ANT
command="cd /data/loadtest/apache-jmeter-2.9/bin;./jmeter -n -t ../../dvdstore-loadtest.jmx -l result.jtl"
If you want to execute the loadtest from the machine that executes the continuous integration, that is you want to make your Jenkins server the load driver, then you might want to use an ANT task for JMeter directly (check http://www.programmerplanet.org/pages/projects/jmeter-ant-task.php for more information)
Step 3: Ensure output format is XML
JMeter results can be formatted as csv or xml file. All reporting tools operate only on xml formats of the jmeter results. Thus you should ensure that you force jmeter to provide the result as xml. Ensure the JMeter property jmeter.save.saveservice.output_format is set to xml (you can define this in the jmeter.properties file). If you forget this step you might encounter errors like “Error! Content is not allowed in prolog.” when generating the report.
Step 4: Integrate in Jenkins
Jenkins already provides the great “Performance” plugin (https://wiki.jenkins-ci.org/display/JENKINS/Performance+Plugin) for reporting JMeter results. Simply activate this plugin (it comes out of the box). I execute loadtests on the weekend, as nobody is working and the hardware is just idling, so this is the perfect time to run a longer loadtest.
I set the build to fail as soon as 40% of the tests fail. Jenkins now automatically builds a performance trend of your application
Automating the execution of non-functional tests is the next logical step to continuous integration. In order to further improve you loadtest you might want to think about additional files that jenkins can tranfer from the testing environment and archive with the job. Thus you have all information available by looking at Jenkins and do not need to connect to the testing environment.