Jenkins Case Study: D4Science

D4Science Amps Up Their Scientific Research Platform With CI/CD Powered By Jenkins​

Submitted By Jenkins User

By Alyssa Tong and Manuele Simi

SUMMARY: D4Science, supporting scientific communities and promoting open science practices by serving 11,000+ registered users in 45 countries, introduced a new delivery pipeline replacing their pre-existing build platform.

CHALLENGE: D4Science needed a cost-effective way to build and release their software framework (gCube) able to support multi-project releases at scale.

SOLUTION: A Continuous Integration/Continuous Delivery (CI/CD) pipeline, scalable, easy to maintain and upgrade at a minimal cost, that represents an innovative approach to software delivering.  

RESULTS

  • Automation of all release activities

  • Fast, incremental releases to quickly address user requirements

  • Build and release software components from 200+ Git repositories within the same day

What do several EU-funded projects, the Food and Agriculture Organization of the United Nations, Mercator Ocean, the European Bioinformatics Institute, CINECA, Sorbonne University, and many other institutions across the world have in common? 

Some of their research and virtual research environments (VREs) are fueled by D4Science, an organization led by the Institute of Information Science and Technologies, an institute of the National Research Council, (ISTI-CNR), located in Pisa, Italy. 

Partially funded by the European Commission, D4Science has created, maintains and constantly enhances an advanced test-bed allowing scientific communities to share knowledge and collaborate in a secure, coordinated, dynamic and cost-effective way.  

Over the years, gCube, D4Science’s software framework, has evolved into a platform where interdisciplinary tools and services can be shared to better serve the collaborative needs of thousands of users worldwide. In fact, nearly ten new EU-funded projects which utilize gCube are currently ongoing or have been approved and are about to start this year. With so many organizations relying on gCube, D4Science decided to migrate to an entirely new platform for controlling, integrating, versioning, and releasing the software. 

image

From one to hundreds of repositories.

Before 2019, D4Science had been using SVN as a version control system with a single repository hosting all the gCube source code. Over time, the issues they were encountering managing their code and releases overcame the benefits SVN was able to provide. 

"gCube is a very extended framework. Composed of hundreds of software components, it has several disparate teams working on various applications and services at the same time," said Manuele Simi, Software Engineer. "Before introducing our new code management platform, each release required an ever-increasing amount of effort, time and resources to coordinate and build. On top of that, our daily work was continually disrupted by recurring problems in the Continuous Integration pipeline that were not under our control."

A first step D4Science took was to migrate from SVN to Git to better manage and version their code. According to Simi, a crucial decision in this transition was how best to map their "huge" SVN repository to Git. After careful consideration, they opted to split the source tree into smaller units of work and create a Git repository for each. With hundreds of repositories to release and a build system that could no longer meet the needs of this demanding scenario, it was clear that they needed to switch to a new Continuous Integration and Continuous Delivery pipeline.

Empowering e-Science and virtual research communities with software released via Jenkins.

The Jenkins Declarative Pipelines implemented by D4Science provide full control of orchestration activities for their complex releases. Additionally, all build jobs are coordinated in a configurable and dynamic fashion. With Jenkins, D4Science has now the capabilities to:

  • build and release software components from 200+ Git repositories within the same day by launching a single pipeline job;

  • reuse the same build jobs across different build stages;

  • require zero effort of the development team at release time;

  • get reliable, fast Continuous Integration feedback   

With minimal effort, D4Science was able to replace the pre-existing build platform that they had relied on for several years. With Jenkins, they gained an easy to maintain, scalable, state-of-the-art CI/CD server that can be upgraded at minimal costs. Most importantly, according to Simi, they solved all the issues that made their previous Continuous Delivery pipeline so expensive.

"With the solutions implemented with Jenkins, we completely reshaped our delivery phase by saving effort, time, and resources," said Simi. "We now have fast, incremental releases managed by a single person, where previously we had a dedicated team just for the release activities."

Dozens of research institutions and organizations worldwide are saving time and resources by employing VREs created with gCube. With the capacity to promptly release new software addressing users' requirements, D4Science continues to receive a stream of projects funded by the European Commission and to attract the interest of diverse communities of practices engaged in research projects, with the promise of more to come.

REFERENCES:

  1. D4Science: https://www.d4science.org/ - @d4science

  2. gCube: https://www.gcube-system.org/ 

  3. ISTI-CNR: https://www.isti.cnr.it/ 

  4. M. Assante,  L. Candela, D. Castelli, R. Cirillo, G. Coro, L. Frosini, L. Lelii, F. Mangiacrapa, P. Pagano, G. Panichi,  F. Sinibaldi, Enacting open science by D4Science, Future Generation Computer System (2019)