Complete Infrastructure Automation Using Jenkins

Background: The LinuxWorld India training program allows students to work on live projects provided by the organization. In this way, they get real exposure to the IT industry. LinuxWorld offers training in the city of Jaipur, with a team of highly experienced and expert trainers. The students can enroll in programs like DevOps, Docker, Splunk, Cloud Computing, Big Data Hadoop, RedHat certified programs, Python, Splunk, python during their training program.

For my project, I wanted to automate my complete process and remove human intervention, which can bring so many errors. Rectifying them will make the process a lot slower and deteriorate the reputation. So, it is imperative to both be free of errors and speed up processes simultaneously. 

Humans make the process slow. They work according to their available time and need. But if a task is assigned to a machine, it will do it when commanded. That is the main reason to choose Jenkins!!

Goals: Creating a faster setup for the development of ML models using Jenkins for auto-error detection and accuracy checks.

Solution & Results

Jenkins is the ultimate CI/CD solution. You name a feature/plugin, and Jenkins has the proper integration and intelligence for it. Jenkins provides the best solution leveraging the Assembly pipeline for any job. I wanted Jenkins to keep track of the accuracy of my ML model. Until the accuracy of a certain percent is achieved, some of the hyperparameters keep on changing based on a python script being run by Jenkins itself. This makes the whole process automated until accuracy is achieved. 

Once the desired percent is achieved, the model converts to an h5 file format where the weights and bias are stored. This model can be used later by providing the required parameters to calculate the given model’s prediction. This whole process – which would take so much time, effort, and sometimes, frustration if a human is involved – is simpler using Jenkins. In the end, after the pipeline has finished computing, an email alert is sent to the admins.

The key features of Jenkins that I used included: Jenkins Assembly Pipeline, Git Integration, the Extended Email Plugin.

The integration is the main crux of the whole project: it just rocks the show. Everything is visual; the pipeline is colorful and complete in its own form. It is great to watch it run and conduct the job on its own. The Groovy-based syntax used to create a Jenkinsfile requires even less from developers and the ops team. 

Finally, the use of the Jenkins tool reduces the build time and makes it as error-prone as possible. Most importantly, it helps to easily shift left. Best of all, testing can be integrated within the pipeline way before the deployment.

My project was successful and enabled many benefits, including: