As an independent open source project, the Jenkins project maintains most of its own infrastructure including services which help to keep the project running. The kind of things that fall into "infrastructure" can span from operating virtual machines, containers, configuring network or developing and maintaining project-specific applications to make the development of Jenkins core and plugins more efficient.
Because we strongly believe in Open Source principles, we also apply them to our infrastructure. As such we consider ourself as an open infrastructure project where everybody is invited to learn, share, contribute.
Jenkins infrastructure is maintained by the Jenkins Infrastructure team. This team consists of Jenkins contributors and volunteers who provide a best-effort support depending on their availability and other commitments. We are always looking for contributors who are interested to improve Jenkins services and to help with maintaining them.
Jenkins Infra team has a weekly meeting on Friday, 14:00 UTC. This event is a regularly scheduled meeting to sync on the various ongoing infrastructure initiatives within the Jenkins project. This meeting is open to everyone who is interested. See the Event Calendar for meeting links.
You can find Infra team meeting agenda and meeting notes on jenkins-infra/documentation.
The Jenkins project runs the most of its infrastructure on Azure, sponsored by Continuous Delivery Foundation. This infrastructure is provisioned from the repository jenkins-infra/azure using Terraform and using AKS charts.
While we try to stick to Azure, we still have machines sponsored by different organizations such as OSUOSL, AWS, Rackspace or CloudBees.
The Jenkins project’s infrastructure is managed and orchestrated via a git repository named jenkins-infra/jenkins-infra using Puppet. We are also using a Kubernetes cluster orchestrated with jenkins-infra/charts using Helm & Helmfiles.
Infrastructure is monitored via Datadog. Datadog configuration is automated with Terraform on jenkins-infra/datadog
We also provide different dashboards to share the health of our systems.
Another major aspect of the Jenkins Infrastructure project, is the maintenance of all services either provided by us or third parties. Here is a non-exhaustive list of services that we provide and maintain.
| Service | JIRA component(s) | Repository and links |
|---|---|---|
|
||
|
||
Infra CI - https://infra.ci.jenkins.io/ |
|
|
Release CI - https://release.ci.jenkins.io/ |
|
|
Private artifacts - https://repo.private.jenkins.io/ |
||
|
||
Keycloak identity management |
||
LDAP |
|
|
SSL certificate renewal |
||
|
||
|
||
Plugin Site API Backend |
|
|
|
||
| Service | JIRA component(s) | Repository and links |
|---|---|---|
|
||
|
||
|
Mirrorbits helm chart, mirror list, mirror stats, Running a mirror |
|
|
old mirrors.jenkins.io Puppet scripts (search for |
|
|
||
Code published with GitHub pages |
||
|
||
VPN |
|
|
Wiki (deprecated) |
|
| Service | Provider | JIRA component(s) | Repository |
|---|---|---|---|
JFrog |
|
- |
|
GitHub |
|
- |
|
Datadog |
|
||
Pagerduty |
- |
||
GitLab |
- |
||
Fastly |
- |
||
Namecheap |
- |
||
Linux Foundation |
- |
Our infrastructure is an open infrastructure project made by and for the Jenkins community. In other words, it’s a contributors driven project. While we can’t share publicly everything like secrets, certificates,… we still try to be as transparent as possible so that everybody can understand and improve our infrastructure without having privileged accesses. If you have any idea that could help the infra or interest the community, feel free to make suggestions.
In order to contribute to infrastructure project, we ask people to follow the next steps
Pick up a task => Communicate => Implement => Deploy => Review
To keep track of the work that needs to be done on the Jenkins infrastructure project, we use a JIRA project named "INFRA" at Jira. If you want to contribute, the first step is to find the JIRA issue in this project that you want to work on, assign it to you, then communicate about it (see Communicate).
If you can’t find an appropriate issue, please create a new one with a clear description:
Why (what is the problem to solve - high level value)?
What (what your proposal to solve the problem)?
How (what are the technical changes to do)?
You may also specify components and finally you can communicate about it, using the suggestions from the next section.
|
Good First Issues
If you want to start contributing on the Jenkins infrastructure,
you can find a list of "first good issues" to be considered (they all have the label |
Before any implementation, it’s important to verify that first, there is (still) a need for some implementation and then that no work has been done in the past. The best way for that is to either look after similar Jira tickets, ask on IRC or on the mailing list. You can also join our weekly meetings to discuss and coordinate the changes.
When the subject is too broad or hard to explain in few lines, we write an IEP document which stand for "Infrastructure Enhancement Proposal", the purpose of this document is to explain why we need something, how we want to solve it, and why we took a final decision. Finally, once you have your ticket id, you can start looking for knowledgeable people.
Anyway keep in mind that it’s always better to have too much information than too little and in the end you’ll probably be the best person who can work on your case.
+----------------------------------+
| |
| Pick up or Create INFRA Ticket |
| |
+----+----+------------------------+
| | If no responses after few days
| | promote it on
| | +------------------------------------------+
| | | |
| +--------------------> IRC: Libera Chat #jenkins-infra <----+
| | | | |
| | +------------------------------------------+ |
| | +------------------------------------------+ |
| | | | |
| +--------------------> Mail: jenkins-infra@googlegroups.com <----+
| | | |
| +------------------------------------------+ |
| If the topic is too big |
| |
| +-------------------------------------------+ |
| | | |
+--------------------> IEP: https://github.com/jenkins-infra/iep |--------+
| |
+-------------------------------------------+
Once there is an agreement about the approach and before any changes, we ask contributors to respect the following rules.
Those rules are just what we consider "best practices" for a contributors driven project and can be adapted depending on specific git repositories.
So it’s easier for everybody to find/review/audit changes and suggest improvements.
So we always have different people who understand infrastructure changes.
Rem: Non regular contributors are more than welcome to share their expertise or just ask question which also help to spot incoherences.
Code Reviews Purpose:
Educate author and the team about the changes in code being made
Discover logic or security issues not covered by tests
Gather improvement suggestion for code readability or efficiency
So we feel more comfortable when merging PRs and we avoid regression issues.
So we only have one source of truth and we don’t break others people works. And if it’s not possible, then it needs to be well communicated and documented cf. rule #1.
Fork project -> Create Feature Branch -> Open Pull Request -> Ask Review -> Merge
The deployment step is the only moment where we need approval from someone with elevated permission. As already mentioned, even if we try to be as open as possible, we don’t want to share privileged accesses with every contributors even if we trust them and that mainly for security reasons.