Custom Distribution Service : Phase 3 Blogpost

Published on 2020-08-31 by Sladyn Nunes

Hello everyone,

This is the final blog post for the Custom Distribution Service project during the Google Summer of Code timeline. I have mixed feelings since we are almost near the finish line for one of the most amazing open source programs out there. However, it is time to wrap things up for this project and achieve a state where the project can be built upon and extended further. This phase has been super busy with respect to the bug fixes, testing and getting the project hosted, so let us get straight into the phase 3 updates.

Fixes and Code quality assurance

Set Jenkinsfile agent to linux

We realised that the build was failing on windows and that there was not really a use-case for running it on windows for right now. Maybe it could be on a future roadmap. Therefore, we decided to shift the testing to only linux agents with respect to running the tests on the jenkins server.

Backend port error message

Spring boot has a default message on the port:8080 and therefore we wanted to change it to a custom message on the backend. So the major takeaway here is that we needed to implement the Error Controller interface and include a custom message in it. This was technical debt from the last phase and was completed and merged during this phase.

  • Pull Request #92

PMD Analysis

In order to enhance the quality of the code, the PMD source code analyser was applied to the project. It helped me catch tons of errors. When the initial PMD check was run and we found approximately 162 PMD errors. We realised some of them were not relevant and some of them could be fixed later.

Findbugs Analysis

Another tool to improve code quality that we included in this phase was findbugs. It did catch around 5-10 bugs in my code which I immediately resolved. Most of them were around the Closeable HTTP Request and an easy fix was the try with resources.

Jacoco Code Coverage

We needed to make sure most of the code we write had proper coverage for all branches and lines. Therefore we decided to include a JaCoco Code Coverage reporter that helped us find the uncovered lines and areas we need to improve coverage on.

Remove JCasC generation

While developing the service we quickly realised that the generation of the war package broke if we included a configuration as code section but did not provide a path to the corresponding required yml file. Therefore we took a decision to remove the casc section all together. Maybe it will comeback in a future patch

  • Pull Request link: #127

  • Issue link: #65

Minor Fixes

  • Logging Fix: #99

  • Docs Fix : link: #120

  • Update Center Dump Fix : link: #125

  • Class Path Fix: link: #126

  • Release Drafter Addition: link: #136

Front end

There was no community configuration link present for navigation which was added here. Now it is easier to navigate to the community page from the home page itself.

Docker updates

Build everything with Docker

This was one of the major changes this phase with respect to making the service very easy to spin up locally, this change will greatly help community adoption since it eliminates the tools one needs to install locally. Initially the process was to run maven locally, generate all of the files and then copy all of its contents into the container. However, with this change we are going to generate all of the files inside the docker container itself. Allowing the user to just run a couple of commands to get the service up and running.

So some of the major changes we did with respect to the dockerfile was:

a) Copy all of the configuration files and pom.xml into the container.

b) Run the command mvn clean package inside the container which generates the jar.

c) Run the jar inside the container.

Hosting updates

This process was supposed to be a future roadmap, however the infra team approved and was super helpful in making this process as smooth as possible. Thanks to Gavin, Tim and Oblak for making this possible. Here is the google group dicussion

The project has now been hosted here as a preview. It still needs some fixes to be fully functional.

  • Infra Docker PR #131

  • Infra Project Addition PR link: #393

Testing Updates

Unit test the services

With respect to community hosting and adoption, testing of the service one of the most important and major milestones for this phase was to test the majority of the code and we have completed the testing with flying colors. All of the services have been completely unit tested, which is a major accomplishment. For the testing of the service we decided to go with wiremock which can be used to mock external services. Kezhi’s comment helped us to understand what we needed to do since he had done something quite similar in his Github Checks API project.

So we basically wiremocked the update-center url and made sure we were getting the accurate response with appropriate control flow logic tested.

wireMockRule.stubFor(get(urlPathMatching("/getUpdateCenter"))
                .willReturn(aResponse()
                        .withStatus(200)
                        .withHeader("Content-Type", "application/json")
                        .withBody(updateCenterBody)));

Add Update Center controller tests

Another major testing change involved testing the controllers. For this we decided to use the wiremock library in java to mock the server response when the controllers were invoked.

For example: If I have a controller that serves in an api called /api/plugin/getPluginList wiremock can be used to stub out its response when the system is under test. So we use something like this to test it out.

 when(updateService.downloadUpdateCenterJSON()).thenReturn(util.convertPayloadToJSON(dummyUpdateBody))

When the particular controller is called the underlying service is mocked and it returns a response according to the one provided by us. To find more details the PR is here.

Add Packager Controller Tests

Along with the update center controller tests another controller that needed to be tested was the packager controller. Also we needed to make sure that all the branches for the controllers were properly tested. Additional details can be found in the PR below.

Docker Compose Tests

One problem that we faced the entire phase was the docker containers. We regularly found out that due to some changes in the codebase the docker container build sometimes broke, or even sometimes the inner api’s seemed to malfunction. In order to counteract that we decided to come up with some tests locally. So what I did was basically introduce a set of bash scripts that would do the following:

a) Build the container using the docker-compose command.

b) Run the container.

c) Test the api’s using the exposed port.

d) Teardown the running containers.

User Documentation

We also included a user docs guide so that it makes it super easy to get started with the service.

Future Roadmap

This has been a super exciting project to work on and I can definitely see this project being built upon and extended in the future.

I would like to talk about some of the features that are left to come in and can be taken up in a future roadmap discussion

a) JCasC Support:

Description: Support the generation of a Jenkins Configuration as Code file asking the user interactively for the plugins they select what would be the configuration they would want eg: If the user selects the slack plugin we need to ask him questions like what is the slack channel? what is the token? etc, and on the basis of this generate a casc file. This feature was initially planned to go into the service but we realised this is a project in its own capacity.

b) Auto Pull Request Creation:

Description: Allow users to create a configuration file and immediately open a pull request on github without leaving the user interface. This was originally planned using a github bot and we started the work on it. But we were in doubt if the service would be hosted or not and therefore put the development on hold. You can find the pull requests here:

  • Github Controller #72

  • Pull Request Creation Functions #66

c) Synergy with Image Controller

Description: This feature requires some planning, some of the questions we can ask are:

a) Can we generate the images (i.e Image Controller). b) Can we have the service as a multipurpose generator ?

Statistics

This phase has been the busiest of all phases and it has involved a lot of work, more than I had initially expected in the phase. Although lines of code added is not an indication of work done, however 800 lines of Code added is a real personal milestone for me.

Pull Requests Opened

26

Lines of Code Added

1096

Lines of Docs Added

200

Gitter Channel Link
GSoC Proposal
Design Document
Daily Notes

About the Author
Sladyn Nunes

Sladyn is a Computer Science student at Mumbai University, India. He is participating in Community Bridge 2019 to provide development tools to JCasC plugin in the form of IDE integrations, schema architecture improvements and configuration extension points Configuration as Code Plugin.