Serverless Builds for Doctors

Background: Carta Healthcare is dedicated to automating and simplifying the work that burns out clinical staff, so they can focus on patient care. When it comes to building the software to do this, DevOps Engineer Vitor Lobao knew the company needed to grow the team so they could have as many parallel builds as they wanted without burning piles of cash. "Our software has an extensive test suite that makes use of several databases," Lobao admitted, "so the queue was long and the machines are huge. Plus, we want them to scale up and down as jobs are triggered or concluded." 

Goals:  Build cost-effective agents for team growth

Solution & Results: Lobao and his team researched solutions but settled on Jenkins. "Every alternative is either limited or a bag of scripts," he said. "But Jenkins was paramount in providing an elegant way of achieving great results."

Lobao chose to deploy a Jenkins master on a Kubernetes cluster and allow it to dispatch agents using pods containing the correct images required for the builds, not have hardcoded credentials, and be entirely IaC. 

Lobao’s DevOps team came up with some creative workarounds when confronted with legacy limitations. "We had to maintain compatibility between the old and the new build servers," he clarifies. "Many folder structures that used to be present locally on the old server were now either on NFS servers or on cloud storage, so we had to fake folder structures symlinking. This was kept until the rollout was complete and the build process was stable." 

"We also used the 'requests' resource section of the pod YAML to reserve the correct amount of memory in a way so that every pod gets its own machine to work," Lobao added. "They get assigned to the correct node pool, and are configured to scale up once resources are requested and to scale down resources are freed."

How does that look to Carta Healthcare? As Lobao explains: 1 new job = 1 pod = 1 machine scaling up. Then when 1 job is done = 1 pod terminated = 1 machine idle, then scaled down. 

"Neat," admits Lobao.

For Lobao, the trickiest part was that the versions that should be used by the pod to test the software were not yet built. "To solve that," he explains, "we had to spin up the agent pod containing placeholder versions, build the images, update the container images on the fly to the appropriate versions, and then run the test suite."

"I didn’t feel like reinventing the wheel," said Lobao. "I felt there was a solid community backing up the project and its extensions." The Carta Healthcare team relied on several Jenkins capabilities, including Pipelines, Credential Binding, Configuration as Code, Bitbucket, Hashicorp Vault, and the Kubernetes plugin, which Lobao calls "the king of them all."

Results included: