Background: We had been testing technology package code but it was done manually and took around 2 to 3 days to achieve. We searched for the best technology to automate the continuous Integration Testing, we came across Jenkins.
The challenges we faced were many. First, the project was quite helpful for Jenkins users to test their technology package being developed, but there was a need to improve on several aspects due to the utility code written in Python for creating a new job or updating an existing job. These are cited below:
Performance - Overnight results were expected from Continuous Integration setup using Jenkins
Reliability - Results should be reliable enough to take further decisions on the package being tested
Stability - Execution status should be stable between different runs
Duration - We wanted testing to do more in less time, which was achieved using the Jenkins parallelization concept
Confidence - Last but not least, we were hoping to build confidence in using Jenkins for management, users, and other stakeholders
Goals: The main goal of this project was to test every new technology package automatically overnight. But we were also interested in:
time efficiency
easy-to-trace build and debug code issues
less effort and complexity
easy-to-add test cases in one place
easy-to-manage project configurations
"Users and stakeholders were not initially interested in using Jenkins due to utility code implementation required to create and update test cases. But by using the Jenkins parallelization concept, we were able to achieve faster releases with no performance, reliability, or stability issues. Management then took the decision to rollout Jenkins in a big way across the teams."
Solution & Results: Using Jenkins, we build a parametrized project that can be used by variant teams. They specify the version of the application they want to develop, and Jenkins takes responsibility to create the technology package code and does the necessary configurations automatically. The customer receives a notification when the new fix has been released and can provide testing without human interaction.
Prior to using Jenkins, testing was carried manually by individual teams. After incorporating Jenkins, we have a centralized place to trigger and monitor the results. And Jenkins maintenance is done by the software 'teams.
Out of all the great results we’ve seen, these are the top four:
Our build and testing time is reduced from 3 days to overnight
We achieved automated continuous integration of technology package testing
We achieved parallel execution with the Jenkins Parallellization concept
We achieved stability between different releases