Parallel stages with Declarative Pipeline 1.2

Published on 2017-09-25 by Andrew Bayer

After a few months of work on its key features, I’m happy to announce the 1.2 release of Declarative Pipeline! On behalf of the contributors developing Pipeline, I thought it would be helpful to discuss three of the key changes.

A Pipeline with Parallel stages

Parallel Stages

First, we’ve added syntax support for parallel stages. In earlier versions of Declarative Pipeline, the only way to run chunks of Pipeline code in parallel was to use the parallel step inside the steps block for a stage, like this:

/* .. snip .. */
stage('run-parallel-branches') {
  steps {
    parallel(
      a: {
        echo "This is branch a"
      },
      b: {
        echo "This is branch b"
      }
    )
  }
}
/* .. snip .. */

While this works, it doesn’t integrate well with the rest of the Declarative Pipeline syntax. For example, to run each parallel branch on a different agent, you need to use a node step, and if you do that, the output of the parallel branch won’t be available for post directives (at a stage or pipeline level). Basically the old parallel step required you to use Scripted Pipeline within a Declarative Pipeline.

But now with Declarative Pipeline 1.2, we’ve introduced a true Declarative syntax for running stages in parallel:

Jenkinsfile
pipeline {
    agent none
    stages {
        stage('Run Tests') {
            parallel {
                stage('Test On Windows') {
                    agent {
                        label "windows"
                    }
                    steps {
                        bat "run-tests.bat"
                    }
                    post {
                        always {
                            junit "**/TEST-*.xml"
                        }
                    }
                }
                stage('Test On Linux') {
                    agent {
                        label "linux"
                    }
                    steps {
                        sh "run-tests.sh"
                    }
                    post {
                        always {
                            junit "**/TEST-*.xml"
                        }
                    }
                }
            }
        }
    }
}

You can now specify either steps or parallel for a stage, and within parallel, you can specify a list of stage directives to run in parallel, with all the configuration you’re used to for a stage in Declarative Pipeline. We think this will be really useful for cross-platform builds and testing, as an example. Support for parallel stages will be in the soon-to-be-released Blue Ocean Pipeline Editor 1.3 as well.

You can find more documentation on parallel stages in the User Handbook.

Defining Declarative Pipelines in Shared Libraries

Until the 1.2 release, Declarative Pipelines did not officially support defining your pipeline blocks in a shared library. Some of you may have tried that out and found that it could work in some cases, but since it was never an officially supported feature, it was vulnerable to breaking due to necessary changes for the supported use cases of Declarative. But with 1.2, we’ve added official support for defining pipeline blocks in src/.groovy files in your shared libraries. Within your src/.groovy file’s call method, you can call pipeline { …​ }, or possibly different pipeline { …​ } blocks depending on if conditions and the like. Note that only one pipeline { …​ } block can actually be executed per run - you’ll get an error if a second one tries to execute!

Major Improvements to Parsing and Environment Variables

Hopefully, you’ll never actually care about this change, but we’re very happy about it nonetheless. The original approach used for actually taking the pipeline { …​ } block and executing its contents was designed almost two years ago, and wasn’t very well suited to how you all are actually using Declarative Pipelines. In our attempts to work around some of those limitations, we made the parsing logic even more complicated and fragile, resulting in an impressive number of bugs, mainly relating to inconsistencies and bad behavior with environment variables.

In Declarative 1.2, we’ve replaced the runtime parsing logic completely with a far more robust system, which also happens to fix most of those bugs at the same time! While not every issue has been resolved, you may find that you can use environment variables in more places, escaping is more consistent, Windows paths are no longer handled incorrectly, and a lot more. Again, we’re hoping you’ve never had the misfortune to run into any of these bugs, but if you have, well, they’re fixed now, and it’s going to be a lot easier for us to fix any future issues that may arise relating to environment variables, when expressions, and more. Also, the parsing at the very beginning of your build may be about 0.5 seconds faster. =)

More to Come!

While we don’t have any concrete plans for what will be going into Declarative Pipelines 1.3, rest assured that we’ve got some great new features in mind, as well as our continuing dedication to fixing the bugs you encounter and report. So please do keep opening tickets for issues and feature requests. Thanks!

About the Author
Andrew Bayer

Andrew was a core committer to Hudson and the author of numerous plugins.