Use Just Enough Pipeline

Perhaps the biggest misstep people make is deciding that they need to write their entire pipeline in a programming language. After all, Pipeline is a domain specific language (DSL). However, that does not mean that it is a general-purpose programming language.

If you treat the DSL as a general-purpose programming language, you are making a serious architectural blunder by doing the wrong work in the wrong place. Remember that the core of Pipeline code runs on the controller. So, you should be mindful that everything you express in the Pipeline domain specific language (DSL) will compete with every other Jenkins job running on the controller.

For example, it’s easy to include a lot of conditionals, flow control logic, and requests using scripted syntax in the pipeline job. Experience tells us this is not a good idea and can result in serious damage to pipeline performance. We’ve actually seen organizations with poorly written Pipeline jobs bring a controller to its knees, while only running a few concurrent builds.

Wait a minute, you might ask, “Isn’t handling code what the controller is there for?” Yes, the controller certainly is there to execute pipelines. But, it’s much better to assign individual steps of the pipeline to command line calls that execute on an agent. So, instead of running a lot of conditionals inside the pipeline DSL, it’s better to put those conditionals inside a shell script or batch file and call that script from the pipeline.

However, this raises another question: “What if I don’t have any agents connected to my controller?” If this is the case, then you’ve just made another bad mistake in scaling Jenkins pipelines. Why? Because the first rule of building an effective pipeline is to make sure you use agents. If you’re using a Jenkins controller and haven’t defined any agents, then your first step should be to define at least one agent and use that agent instead of executing on the controller.

For the sake of maintaining scalability in your pipeline, the general rule is to avoid processing any workload on your controller. If you’re running Jenkins jobs on the controller, you are sacrificing controller performance. So, try to avoid using Jenkins controller capacity for things that should be passed off to an agent. Then, as you grow and develop, all of your work should be running agents. This is why we always recommend setting the number of executors on the master to zero.

All of this serves to highlight our overarching theme of “using just enough pipeline.” Simply put, you want to use enough code to connect the pipeline steps and integrate tools – but no more than that. Limit the amount of complex logic embedded in the Pipeline itself (similarly to a shell script), and avoid treating it as a general-purpose programming language. This makes the pipeline easier to maintain, protects against bugs, and reduces the load on controllers.

Another best practice for keeping your pipeline lean, fast, and scalable is to use declarative syntax instead of scripted syntax for your Pipeline. Declarative naturally leads you away from the kinds of mistakes we’ve just described. It is a simpler expression of code and an easier way to define your job. It’s computed at the startup of the pipeline instead of executing continually during the pipeline.

Therefore, when creating a pipeline, start with declarative, and keep it simple for as long as possible. Anytime a script block shows up inside of a declarative pipeline, you should extract that block and put it in a shared library step. That keeps the declarative pipeline clean. By combining declarative with a shared library, that will take care of the vast majority of use cases you’ll encounter.

That said, it’s not accurate to say that declarative plus a shared library will solve every problem. There are cases where scripted is the right solution. However, declarative is a great starting point until you discover that you absolutely must use scripted.

Just remember, at the end of the day, you’ll do well to follow the adage: “Use just enough pipeline and no more.”

Mark and Darin both work for CloudBees. CloudBees helps Fortune 1000 enterprises manage and scale Jenkins. Thanks to CloudBees for sponsoring the creation of this blog post.

Mark and Darin joined Hope Lynch and Joost van der Griendt to share additional topics in a CloudBees on-demand recording, "Optimizing Jenkins for the Enterprise". Register for the on-demand recording to receive more information on configuration as code, plugin management, and Pipelines.