Multi-branch Pipeline Jobs Support for GitLab SCM

Published on 2019-06-29 by Parichay Barpanda

This is one of the Jenkins project in GSoC 2019. We are working on adding support for Multi-branch Pipeline Jobs and Folder Organisation in GitLab. The plan is to create the following plugins:

  • GitLab API Plugin - Wraps GitLab Java APIs.

  • GitLab Branch Source Plugin - Contains two packages:

    • io.jenkins.plugins.gitlabserverconfig - Manages server configuration and web hooks management. Ideally should reside inside another plugin with name GitLab Plugin. In future, this package should be moved into a new plugin.

    • io.jenkins.plugins.gitlabbranchsource - Adds GitLab Branch Source for Multi-branch Pipeline Jobs (including Merge Requests) and Folder organisation.

Present State

  • FreeStyle Job and Pipeline(Single Branch) Job are fully supported.

  • Multi-branch Pipeline Job is partially supported (no MRs detection).

  • GitLab Folder Organisation is not supported.

Goals of this project

  • Implement a lightweight GitLab Plugin that depends on GitLab API Plugin.

  • Follow convention of 3 separate plugins i.e. GitLab Plugin, GitLab API Plugin, GitLab Branch Source Plugin.

  • Implement GitLab Branch Source Plugin with support for Multi-branch Pipeline Jobs.

  • Support new Jenkins features such as Jenkins Code as Configuration (JCasC), Incremental Tools.

  • Clear & Efficient design.

  • Support new SCM Trait APIs.

  • Support Java 8 and above.

Building the plugin

No binaries are available for this plugin as the plugin is in the very early alpha stage, and not ready for the general public quite yet. If you want to jump in early, you can try building it yourself from source.

Installation:

  • Checkout source code to your local machine:

git clone https://github.com/baymac/gitlab-branch-source-plugin.git

cd gitlab-branch-source-plugin

  • Install the plugin:

mvn clean install

mvn clean install -DskipTests # to skip tests

  • Run the plugin:

mvn hpi:run # runs a Jenkins instance at localhost:8080

mvn hpi:run -Djetty.port=<port> # to run on your desired port number

If you want to test it with your Jenkins server, after mvn clean install follow these steps in your Jenkins instance:

  1. Select Manage Jenkins

  2. Select Manage Plugins

  3. Select Advanced tab

  4. In Upload Plugin section, select Choose file

  5. Select $<root_dir>/target/gitlab-branch-source.hpi

  6. Select Upload

  7. Select Install without restart

Usage

Assuming plugin installation has done been already.

Setting up GitLab Server Configuration on Jenkins

  1. On jenkins, select Manage Jenkins

  2. Select Configure System

  3. Scroll down to find the GitLab section

    gitlab-section

  4. Select Add GitLab Server | Select GitLab Server

  5. Now you will now see the GitLab Server Configuration options.

    gitlab-server

    There are 4 fields that needs to be configured:

    • Name - Plugin automatically generates an unique server name for you. User may want to configure this field to suit their needs but should make sure it is sufficiently unique. We recommend to keep it as it is.

    • Server URL - Contains the URL to your GitLab Server. By default it is set to "https://gitlab.com". User can modify it to enter their GitLab Server URL e.g. https://gitlab.gnome.org/, http://gitlab.example.com:7990. etc.

    • Credentials - Contains a list of credentials entries that are of type GitLab Personal Access Token. When no credential has been added it shows "-none-". User can add a credential by clicking "Add" button.

    • Web Hook - This field is a checkbox. If you want the plugin to setup a webhook on your GitLab project(s) related jobs, check this box. The plugin listens to a URL for the concerned GitLab project(s) and when an event occurs in the GitLab Server, the server sends an event trigger to the URL where the web hook is setup. If you want continuous integration (or continuous delivery) on your GitLab project then you may want to automatically set it up.

  6. Adding a Personal Access Token Credentials (To automatically generate Personal Access Token see next section):

    1. User is required to add a GitLab Personal Access Token type credentials entry to securely persist the token inside Jenkins.

    2. Generate a Personal Access Token on your GitLab Server:

      1. Select profile dropdown menu from top-right corner

      2. Select Settings

      3. Select Access Token from left column

      4. Enter a name | Set Scope to api,read_user, read_repository

      5. Select Create Personal Access Token

      6. Copy the token generated

    3. Return to Jenkins | Select Add in Credentials field | Select Jenkins

    4. Set Kind to GitLab Personal Access Token

    5. Enter Token

    6. Enter a unique id in ID

    7. Enter a human readable description

    8. Select Add

      gitlab-credentials

  7. Testing connection:

    1. Select your desired token in the Credentials dropdown

    2. Select Test Connection

    3. It should return something like Credentials verified for user <username>

  8. Select Apply (at the bottom)

  9. GitLab Server is now setup on Jenkins

Creating Personal Access Token within Jenkins

Alternatively, users can generate a GitLab Personal Access Token within Jenkins itself and automatically add the GitLab Personal Access Token credentials to Jenkins server credentials.

  1. Select Advanced at the bottom of GitLab Section

  2. Select Manage Additional GitLab Actions

  3. Select Convert login and password to token

  4. Set the GitLab Server URL

  5. There are 2 options to generate token;

    1. From credentials - To select an already persisting Username Password Credentials or add an Username Password credential to persist it.

    2. From login and password - If this is a one time thing then you can directly enter you credentials to the text boxes and the username/password credential is not persisted.

  6. After setting your username/password credential, select Create token credentials.

  7. The token creator will create a Personal Access Token in your GitLab Server for the given user with the required scope and also create a credentials for the same inside Jenkins server. You can go back to the GitLab Server Configuration to select the new credentials generated (select "-none-" first then new credentials will appear). For security reasons this token is not revealed as plain text rather returns an id. It is a 128-bit long UUID-4 string (36 characters).

    gitlab-token-creator

Configuration as Code

No need for messing around in the UI. Jenkins Configuration as Code (JCasC) or simply Configuration as Code Plugin allows you to configure Jenkins via a yaml file. If you are a first time user, you can learn more about JCasC here.

Add configuration YAML:

There are multiple ways to load JCasC yaml file to configure Jenkins:

  • JCasC by default searches for a file with the name jenkins.yaml in $JENKINS_ROOT.

  • The JCasC looks for an environment variable CASC_JENKINS_CONFIG which contains the path for the configuration yaml file.

    • A path to a folder containing a set of config files e.g. /var/jenkins_home/casc_configs.

    • A full path to a single file e.g. /var/jenkins_home/casc_configs/jenkins.yaml.

    • A URL pointing to a file served on the web e.g. https://<your-domain>/jenkins.yaml.

  • You can also set the configuration yaml path in the UI. Go to <your-jenkins-domain>/configuration-as-code. Enter path or URL to jenkins.yaml and select Apply New Configuration.

An example of configuring GitLab server via jenkins.yaml:

credentials:
  system:
    domainCredentials:
      - credentials:
          - gitlabPersonalAccessToken:
              scope: SYSTEM
              id: "i<3GitLab"
              token: "XfsqZvVtAx5YCph5bq3r" # gitlab personal access token

unclassified:
  gitLabServers:
    servers:
      - credentialsId: "i<3GitLab"
        manageHooks: true
        name: "gitlab.com"
        serverUrl: "https://gitlab.com"

For better security, see handling secrets section in JCasC documentation.

Future Scope of work

The second phase of GSoC will be utilized to develop GitLab Branch Source. The new feature is a work in progress, but the codebase is unstable and requires lot of bugfixes. Some features like Multibranch Pipeline Jobs are functioning properly. More about it at the end of second phase.

Issue Tracking

This project uses Jenkins JIRA to track issues. You can file issues under gitlab-branch-source-plugin component.

Acknowledgements

This plugin is built and maintained by the Google Summer of Code (GSoC) Team for Multi-branch Pipeline Support for GitLab. A lot of inspiration was drawn from GitLab Plugin, Gitea Plugin and GitHub Plugin.

Our team consists of: baymac, LinuxSuRen, Marky, Joseph, Justin, Jeff.

With support from: Oleg, Greg, Owen.

Also thanks to entire Jenkins community for contributing with technical expertise and inspiration.

About the Author
Parichay Barpanda

GSoC 2019 student under Jenkins project. Maintainer of GitLab Branch Source Plugin. Software Developer at Udaan, India.