The Silence of the Lambs: Inspecting binaries with Jenkins

Security is an overloaded term with varying meaning in different contexts. For this contribution, I consider security as the sum of rules regarding vulnerabilities (Common Vulnerability and Exposure, CVE), in binaries. In a past blog post, we’ve identified SonarQube already, as a very helpful tool to identify flaws in source code, particularly concerning reliability (bugs), vulnerabilities (security, e.g. CWE, that is common weakness enumaration, and OWASP, that is the Open Web Application Security Project), and maintainability (code smells). Now it is a good time to add another tool to the chain, that is Twistlock, for inspection binaries for security issues. Features of Twistlock include

The underlying use case can be derived from several real-world security initiatives, in enterprises, based on given containerized applications. In practice, it is not a surprise that after adding such new quality gates, you identify historically grown issues. However, there are many good reasons to do so. You don’t need any Word documents to check any governance criteria manually, rather execution and reporting are done automatically and also part of the actions are taken automatically. And above all, of course, your application is quality assured regarding known vulnerability issues, aligned with the DevOps approach: development is interested in quick feedback whether their change would introduce any vulnerabilities, and operations is interested in insights whether and how running applications are affected if a new CVE is discovered.

Conceptually, container inspection is now part of the delivery pipeline and Twistlock processing is now triggered once we have produced our Docker images, see below, in order to get fast feedback.

Software is staged over different environments by configuration, without rebuilding. All changes go through the entire staging process, although defined exception routines may be in place, for details see Michael Hüttermann, Agile ALM (Manning, 2012). The staged software consists of all artifacts which make up the release, consistently, including the business application, test cases, build scripts, Chef cookbooks, Dockerfiles, Jenkins files to build all that in a self-contained way, for details see Michael Hüttermann, DevOps for Developers (Apress, 2012).

In our sample process, we produce a web application that is packaged in a Docker image. The produced Docker images are distributed only if the dedicated quality gate passes. A quality gate is a stage in the overall pipeline and a sum of defined commitments, often called requirements, the unit of work must pass. In our case, the quality gate comprises inspection of produced binaries and it fails if vulnerabilities of severity 'critical' are found. We can configure Twistlock according to our requirements. Have a look how we’ve integrated it into our Jenkins pipeline, with focus on detecting vulnerabilities.

Now let’s start with the first phase to bring our application in shape again, that is gaining insight about the security related flaws.

After we’ve introduced the new quality gate, it failed, see image above. As integration with other tools, Jenkins is the automation engine and does provide helpful context information, however, those cannot replace features and data the dedicated, triggered tool does offer. Thus, this is the moment to switch to the dedicated tool, that is Twistlock. Opening the dashboard, we can navigate to the Jenkins build jobs, that is the specific run of the build, and the respective results of the Twistlock analysis. What we see now is a list of vulnerabilities, and we need to fix those of severity critical in order to pass the quality gate, and get our changes again toward production. The list shows entries of type jar, that is a finding in a binary as part of the Docker image, in our case the WAR file we’ve deployed to a web container (Tomcat), and of type OS, those are issues of the underlying image itself, the operating system, either part of the base image, or as a package added/changed in our Dockerfile.

We can now easily zoom in and examine the vulnerabilities of the Docker layers. This really helps to structure work and identify root causes. Since, typically, a Docker image extends a Docker base image, the findings in the base image are shown on the top, see next screenshot, grouped by severity.

Other Docker layers were added to the base image, and those can add vulnerabilities too. In our case, the packaged WAR file obviously contains a vulnerability. The next image shows how we examine that finding, while this time expanding the Twistlock wizard (that is the plus sign) to directly see the list of found vulnerabilities.

Finding and visualizing the issues are a very good first step, and we’ve even made those findings actionable, so we now have to take action and address them.

To address the findings, we need to split our initiative into two parts:

Let’s proceed bottom up, first coping with the Docker base image.

Let’s now have a look at the used Docker image by looking at the used Dockerfile.

By checking available versions of the official OpenJDK Alpine image, we see that there’s a newer version 8u181 which we could use. We can zoom in and study release notes and contents, or we just pragmatically switch the base image to a more recent version. Often it is a good idea to upgrade versions regularly, in defined intervals. This leads to the following change in the Dockerfile.

There are more options available to fix the issues, but let’s proceed to the second part, the vulnerabilities in the deployment unit.

Before we push this change to GitHub, we also address the vulnerability issue in the deployment unit, that is jetty-io. Here we are a bit unsure about why, in this specific use case, the library is used. To retrieve more information about dependencies, we run a dependency:tree command on our Maven based project. We now see that jetty-io is transitively referenced by org.seleniumhq.selenium:htmlunit-driver. We can surely discuss why this is a compile dependency and the libraries are shipped as part of the WAR, but let’s consider this to be given according to requirements, thus we must take special attention now to version 2.29.0 of the specific library.

Also here we can browse release notes and content (particularly how those libs are built themselves), and come to the conclusion to switch from the used version, that is 2.29.0, to a newer version of htmlunit-driver, that is 2.31.1.

OK, now we are done. We push the changes to GitHub, and our GitHub webhook directly triggers the workflow. This time the quality gate passes, so it looks like our fixes did address the root causes and eliminated those with the configured threshold severity.

Finally, after running through our entire workflow, that is made up of different pipelines, our inspected and quality assured container does successfully run in our production runtime environment, that is on Oracle Cloud.

Crisp, isn’t it?

This closes our quick walkthrough of how to inject security related quality gates into a Jenkins based delivery pipeline. We’ve discussed some concepts and how this can look like with sample tools. In the center of our efforts, we used Jenkins, the swiss army knife of automation. We enriched our ecosystem by integrating couple of platforms and tools, above all Twistlock. After this tasty appetizer you are ready to assess your own delivery pipelines, concepts and tools, and to possibly invest even more attention to security.