Pipeline as YAML: Alpha release

Published on 2018-07-17 by Abhishek Gautam

About me

I am Abhishek Gautam, 3rd year student from Visvesvaraya National Institute of technology, India, Nagpur. I was a member of ACM Chapter and Google student developer club of my college. I am passionate about automation.

Project Summary

This is a GSoC 2018 project.

This project aims to develop a pull request Job Plugin. Users should be able to configure job type using YAML file placed in root directory of the Git repository being the subject of the pull request. The plugin should interact with various platforms like Bitbucket, Github, Gitlab, etc whenever a pull request is created or updated.

Plugin detects the presence of certain types of reports at conventional locations, and publish them automatically. If the reports are not present at their respective conventional location, the location of the report can be configured in the YAML file.

My mentors are Oleg Nenashev (Org Admin), Martin d’Anjou, Kristin Whetstone, Jeff Knurek

Benefits to the community

  • Project administrators will be able to handle pull request builds more easily.

  • Build specifications for pull requests can be written in a concise declarative format.

  • Build reports will be automatically published to Github, Bitbucket, etc.

  • Build status updates will be sent to git servers automatically.

  • Users will not have to deal with pipeline code.

  • If there will be no merge conflicts or build failures, the PR can be merged into target branch.

Phase 1 blog post

Please see Phase 1 blog post

Implementations till now

Alpha version of the plugin is released. It supports all features of Multi-Branch Pipeline and offers the following features.

Build description is defined via YAML file stored within the SCM repo. This plugin will depend on GitHub plugin, Bitbucket plugin, Gitlab plugin if users will be using respective platforms for their repositories.

  1. Conversion of YAML to Declarative Pipeline: A class YamlToPipeline is written which will load the "Jenkinsfile.yaml" and make use of PipelineSnippetGenerator class to generate Declarative pipeline code.

  2. Reporting of results, only xml report types is supported for now.

  3. Use of Yaml file (Jenkinsfile.yaml) from target branch.

  4. Git Push step: To push the changes of pull request to the target branch. This is implemented using git-plugin, PushCommand is used for this from git-plugin. credentialId, branch name and repository url for interacting with Github, Bitbucket, etc will be taken automatically from "Branch-Source" (Users have to fill this details of branch source in job configuration UI). (You can see How to run the demo)

  5. StepConfigurator: To generate pipeline code for all supported steps in Jenkins. This is using Jenkins configuration-as-code plugin (JCasC plugin) to configure a particular step object and then that step object is passed to Snippetizer.object2Groovy() method to generate the script of that step.

Jenkinsfile.yaml example

For the phase 1 prototype demonstration, the following yaml file was used. Note that this format is subject to change in the next phases of the project, as we formalise the yaml format definition.

#  Docker image agent example
agent:
 label: my_label
 customWorkspace: path_to_workspace
 dockerImage: maven:3-alpine
 args: -v /tmp:/tmp

  tools:
    maven : maven_3.0.1
    jdk : jdk8

configuration:
  # Push PR changes to the target branch if the build succeeds.
  # default value is false
  pushPrOnSuccess: false

  # Trusted user to approve pull requests
  prApprovers:
    - username1
    - username2
    - username3

environment:
  variables:
    variable_1: value_1
    variable_2: value_2

  # Credentials contains only two fields. credentialId must be present in the Jenkins Credentials
  credentials:
    - credentialId : fileCredentialId
      variable : FILE

      # In user scripts Username and Password can be accessed by LOGIN_USR and LOGIN_PSW
      # respectively as environment variales
    - credentialId : dummyGitRepo
      variable : LOGIN

stages:
  - name: stage1
    agent: any
    steps:
      - sh: "scripts/hello"
      - sleep:
          time: 2
          unit: SECONDS
      - sleep: 2
      - junit:
          testResults: "target/**.xml"
          allowEmptyResults: true
          testDataPublishers:
            - AutomateTestDataPublisher
            - JunitResultPublisher:
                urlOverride: "urlOverride"
    # Post section for "stage1". All Conditions which are available in Jenkins
    # declarative pipeline are supported
    post:
      failure:
        - sh: "scripts/hello"

# Outer post section. Just like declarative pipeline.
post:
  always:
    - sh: "scripts/hello"

Coding Phase 2 plans (Completed)

  • Decide a proper YAML format to use for Jenkinsfile.yaml

  • Create Step Configurator for SPRP plugin. JENKINS-51637. This will enable users to use Pipeline steps in Jenkinsfile.yaml.

  • Automatic indentation generation in the generated PipelineSnippetGenerator class.

  • Write tests for the plugin.

Coding Phase 3 plans

  1. Test Multi-Branch Pipeline features support:

    1. Support for webhooks (JENKINS-51941)

    2. Check if trusted people have approved a pull request and start build accordingly (JENKINS-52517)

  2. Finalize documentation (JENKINS-52518)

  3. Release 1.0 (JENKINS-52519)

  4. Plugin overview blog post

Coding Phase 3 plans after release

  1. Support the “when” Declarative Pipeline directive (JENKINS-52520)

  2. Nice2have: Support hierarchical report types (JENKINS-52521)

  3. Add unit tests, JenkinsRule tests, and ATH tests (JENKINS-52495, JENKINS-52496)

  4. Automatic Workspace Cleanup when PR is closed (JENKINS-51897)

  5. Refactor snippet generator to extensions (JENKINS-52491)

Phase 3 Jira Epic

Phase 2 evaluation presentation video

Video:

Phase 2 evaluation presentation slides

How to reach me

References

About the Author
Abhishek Gautam

Abhishek is a 3rd year Computer Science student from Visvesvaraya National Institute of Technology, Nagpur, India. He has done some website projects for his college technical festival. He is also a regular competitive programmer (abhishekg1128 at codechef). He has done two internships as a Game Programmer as well. He was a member of ACM Chapter and Google student developer club of his college. His interest in automation motivated his participation in the Jenkins GSOC 2018 program.