Machine Learning Plugin project - coding phase 1 blog post

Published on 2020-06-30 by Loghi Perinpanayagam
jenkins gsoc logo small

Welcome back !

This blog post is briefing my coding phase 1 in Jenkins Machine Learning Plugin for this GSoC 2020.

After a fresh introduction of community bonding, On June 1st, coding of GSoC had started officially with phase 1. At this point, every GSoC student should be expected to have a rigid plan with their entire project. With the guidance of mentors I was able to complete a design document and timeline which can be slightly adjustable during the coding. The coding phase was more about coding and discussion.

Quick review

Pull Requests

21

Jira Issues

11

Major Tasks

3

Completed

3

In progress

0

Week 1

I have to ensure that I have a solid architecture for implementing the core of this plugin such that perhaps I or future community will be able to develop R and Julia kernels for this plugin. Factory method design patterns are suitable when users need different types of products ( Python, R and Julia) without knowing much about the internal infrastructure ( Manager of these interpreters ).

All the base classes were implemented this week.

  • Design the Kernel connectors

  • Initiate the interpreter

  • Close the connection

  • Add simple tests

  • Update pom.xml

More than these changes, repo was updated with pull request template and licence header. Readme was extended a little at the end of the week.

Issues and Challenges

  • Git rebase and squash

  • Tests invokes ipython client in the server failed during the CI build

Week 2

With the help of a design document, I had a plan to do the configurations globally and using the Abstract Folder property I could save the configuration and retrieve for the job configuartion. I used to reference some other well developed plugin for the structure of code. That helped me a lot while I was coding. Our first official contributor has popped out his pull request.

Form validations and helper html will be a great help in the user point of view as well as developers. A minor bug was fixed with the guidance of mentors by writing tests with ‘Jenkins WebClient`. Until the end of the week, builder class of the plugin has been implemented with lots of research and discussion. Finally, Test connection was added to the global configuration page to start the connection and test it. A single issue that blocked me using py4j authentication about zeppelin-python was reported in Jira.

Server Configuration

global_config

Issues and challenges

  • Backend depends on Apache zeppelin-python API to connect IPython

  • Find relevant extension points to extend the plugin

Week 3

Earlier in this week, we were trying to merge our IPython builder PR without any memory leaks or bugs that will cause the system to be devastating while running this plugin. Later, this whole week I was implementing a file parser that could copy the necessary files and had the ability to accomplish the file conversion.

Supported file types

  • Python (.py)

  • JSON (Zeppelin notebooks format)

IPython builder was able to run Jupyter Notebooks and Zeppelin formatted JSON files at the end of the 3rd week. Minor issues were fixed in the code. We used ANSI color plugin to fix the abnormal view of error messages produced by the ipython kernel.

Copying and converting Jupyter Notebook

file_convert

Issues and Challenges

Python error messages could not be displayed in rich format If a job is running at user level, but if the python code access file/file path which is not authorized to the user, it returns a permission denied message. While running on agent, notebook has to be written/copied to agent workspace Artifacts should be maintained/reachable from controller after build.

Week 4

As all the major tasks has done, the demo preparation and plan for a experimental release was carried during the last week. There were lots of research on how to connect to a existing kernel in remote. Demo and presentation were prepared along the week.

Issues and Challenges

  • Releasing the first version was bit late

Knowledge transfer

How to debug the code through IntelliJ

  • Edit configuration → Add new Configuration → Maven

  • Command line → type hpi:run

  • Click the debug icon on the toolbar or go to Run menu then Debug

How to setup to test the plugin

  • Setup JDK 8 and Maven 3.5.*

  • Create a directory $ mkdir machine-learning-plugin

  • Create a virtual environment $ virtualenv venv

  • Activate your virtual environment $ source venv/bin/activate

  • Run $ which python to ensure your python path

  • $ git clone https://github.com/jenkinsci/machine-learning-plugin.git

  • Run $ mvn clean install from the machine-learning-plugin directory

  • Run $ mvn hpi:run to start Jenkins with the plugin

  • Set up the builder with localhost and other parameters

  • Create a job

  • Write python code like print(“plugin works”)

  • Build the job

Issues and bugs

About the Author
Loghi Perinpanayagam

Computer Science and Engineering Student at University of Moratuwa, Sri Lanka. He has been selected for Machine Learning plugin for Data Science in GSoC 2020 for Jenkins project. Hightly interested and contributing on open source projects.