Controlling the Flow with Stage, Lock, and Milestone

Published on 2016-10-16 by Patrick Wolf
This is a guest post by Patrick Wolf, Director of Product Management at CloudBees.

Recently the Pipeline team began making several changes to improve the stage step and increase control of concurrent builds in Pipeline. Until now the stage step has been the catch-all for functionality related to the flow of builds through the Pipeline: grouping build steps into visualized stages, limiting concurrent builds, and discarding stale builds.

In order to improve upon each of these areas independently we decided to break this functionality into discrete steps rather than push more and more features into an already packed stage step.

  • stage - the stage step remains but is now focused on grouping steps and providing boundaries for Pipeline segments.

  • lock - the lock step throttles the number of concurrent builds in a defined section of the Pipeline.

  • milestone - the milestone step automatically discards builds that will finish out of order and become stale.

Separating these concerns into explicit, independent steps allows for much greater control of Pipelines and broadens the set of possible use cases.

Stage

The stage step is a primary building block in Pipeline, dividing the steps of a Pipeline into explicit units and helping to visualize the progress using the "Stage View" plugin or "Blue Ocean". Beginning with version 2.2 of "Pipeline Stage Step" plugin, the stage step now requires a block argument, wrapping all steps within the defined stage. This makes the boundaries of where each stage begins and ends obvious and predictable. In addition, the concurrency argument of stage has now been removed to make this step more concise; responsibility for concurrency control has been delegated to the lock step.

stage('Build') {
  doSomething()
  sh "echo $PATH"
}

Omitting the block from stage and using the concurrency argument are now deprecated in Pipeline. Pipelines using this syntax will continue to function but will produce a warning in the console log:

Using the 'stage' step without a block argument is deprecated

This message is only a reminder to update your Pipeline scripts; none of your Pipelines will stop working. If we reach a point where the old syntax is to be removed we will make an announcement prior to the change. We do, however, recommend that you update your existing Pipelines to utilize the new syntax.

note: Stage View and Blue Ocean will both work with either the old stage syntax or the new.

Lock

Rather than attempt to limit the number of concurrent builds of a job using the stage, we now rely on the "Lockable Resources" plugin and the lock step to control this. The lock step limits concurrency to a single build and it provides much greater flexibility in designating where the concurrency is limited.

  • lock can be used to constrain an entire stage or just a segment:

stage('Build') {
  doSomething()
  lock('myResource') {
    echo "locked build"
  }
}

  • lock can be also used to wrap multiple stages into a single concurrency unit:

lock('myResource') {
  stage('Build') {
    echo "Building"
  }
  stage('Test') {
    echo "Testing"
  }
}

Milestone

The milestone step is the last piece of the puzzle to replace functionality originally intended for stage and adds even more control for handling concurrent builds of a job. The lock step limits the number of builds running concurrently in a section of your Pipeline while the milestone step ensures that older builds of a job will not overwrite a newer build.

Concurrent builds of the same job do not always run at the same rate. Depending on the network, the node used, compilation times, test times, etc. it is always possible for a newer build to complete faster than an older build. For example:

  • Build 1 is triggered

  • Build 2 is triggered

  • Build 2 builds faster than Build 1 and enters the Test stage sooner.

Rather than allowing Build 1 to continue and possibly overwrite the newer artifact produced in Build 2, you can use the milestone step to abort Build 1:

stage('Build') {
  milestone()
  echo "Building"
}
stage('Test') {
  milestone()
  echo "Testing"
}

When using the input step or the lock step a backlog of concurrent builds can easily stack up, either waiting for user input or waiting for a resource to become free. The milestone step will automatically prune all older jobs that are waiting at these junctions.

milestone()
input message: "Proceed?"
milestone()

Bookending an input step like this allows you to select a specific build to proceed and automatically abort all antecedent builds.

milestone()
lock(resource: 'myResource', inversePrecedence: true) {
  echo "locked step"
  milestone()
}

Similarly a pair of milestone steps used with a lock will discard all old builds waiting for a shared resource. In this example, inversePrecedence: true instructs the lock to begin most recent waiting build first, ensuring that the most recent code takes precedence.

Putting it all together

Each of these steps can be used independently of the others to control one aspect of a Pipeline or they can be combined to provide powerful, fine-grained control of every aspect of multiple concurrent builds flowing through a Pipeline. Here is a very simple example utilizing all three:

stage('Build') {
  // The first milestone step starts tracking concurrent build order
  milestone()
  node {
    echo "Building"
  }
}

// This locked resource contains both Test stages as a single concurrency Unit.
// Only 1 concurrent build is allowed to utilize the test resources at a time.
// Newer builds are pulled off the queue first. When a build reaches the
// milestone at the end of the lock, all jobs started prior to the current
// build that are still waiting for the lock will be aborted
lock(resource: 'myResource', inversePrecedence: true){
  node('test') {
    stage('Unit Tests') {
      echo "Unit Tests"
    }
    stage('System Tests') {
      echo "System Tests"
    }
  }
  milestone()
}

// The Deploy stage does not limit concurrency but requires manual input
// from a user. Several builds might reach this step waiting for input.
// When a user promotes a specific build all preceding builds are aborted,
// ensuring that the latest code is always deployed.
stage('Deploy') {
  input "Deploy?"
  milestone()
  node {
    echo "Deploying"
  }
}

For a more complete and complex example utilizing all these steps in a Pipeline check out the Jenkinsfile provided with the Docker image for demonstrating Pipeline. This is a working demo that can be quickly set up and run.

About the Author
Patrick Wolf

This author has no biography defined. See social media links referenced below.