Jenkins is the way to to produce ultra-modern and sophisticated electronic devices

Apple: Helping workforce of 5000 developers

Submitted By Jenkins User Ankit Menon
he DevOps team at Apple had to cope with a high frequency of simultaneous builds which affected resources and testing.
Organization: Apple, apple.com
Industries: Consumer Electronics
Programming Languages: Java, Python
Community Support: Jenkins Users Google Group or IRC Chat, Jenkins.io websites & blogs, Spoke with colleagues and peers

Jenkins — paired with Kubernetes — allows Apple DevOps to always stay ahead of the game.

Background:  Since there are more than 5,000 developers working on the same project simultaneously, the main challenge was to cope with the high frequency of builds every minute.  We also had to tackle the resource consumption on the Jenkins machine and the nodes: there are more than 4,000 nodes running across five Jenkins controllers hosted on Kube. 

Goals: Making the rollout of our cellular platform software quick and easy, while automating test runs.

"Jenkins speeds up the process of development which makes the software run better with the hardware. It’s the secret of successful rollouts by giving us the ability to run extensive automated tests."
image— Ankit Menon, Senior DevOps Engineer, Apple

Solution & Results: We moved the entire node mechanism to Kube. Since Jenkins was already running on Kube, we came up with a solution to create nodes on-demand according to the build queue and load. In this way, there are no build queues standing in a 'pending' state. 

We are using scalable resources for the Kube clusters and for hosting Jenkins, so resources like Print Via Computer (PVC), never run short in case of heavy load. Also, monitoring was super easy since it got coupled nicely with Prometheus and Grafana, which have awesome dashboards and alert tools, so we are always ahead of any unforeseen disasters.

Jenkins has awesome plugins. One of the most important is the Kubernetes plugin, which allowed the builds to run in a container and in a scalable way. We solved the problem of the increasing number of build nodes by creating a mechanism of spinning up nodes on-demand on Kube using custom-built images. 

Also the "throttle concurrent builds plug-in" came to the rescue before implementing our recent nodes on Kube setup by limiting the incoming builds for more efficient throughput.

The results are better than expected:

  • Handles thousands of check ins from dev which speeds up merges

  • Build times are a lot faster with the new node mechanism 

  • Much better results at the end of every quarter with continuous automated testing