Machine Learning Plugin project - Coding Phase 3 blog post

Published on 2020-08-27 by Loghi Perinpanayagam
jenkins gsoc logo small

Good to see you all again !

This is my final blog post about coding phase 3 in Jenkins Machine Learning Plugin for GSoC 2020. Being at the end of GSoC 2020, we had to finish all the pending issues and testing before a stable release in the main repository. Throughout this program, there were lots of learning and hard work will make this plugin valuable to the Data Science and Jenkins community.

Summary

With combining all of the work in phase 1, 2 and 3, initial version of Machine learning plugin( 1.0 ) was successfully released in Jenkins plugin repository. An interesting feature which allows users to connect to their existing programming language kernels more than connecting to only IPython kernel was introduced in this phase. It can be selected in multiple steps with different kernel. Images and graphs produced by Jupyter notebooks will be saved in user preferred folder in the workspace that can be used for reporting/analytic purposes later. Hoping this blog summarizes the Machine Learning’s features and future contributions. Thank you for your interest and support !!!

Main features of Machine Learning plugin

  • Execute Jupyter notebooks directly

  • Run different language scripts using multiple build steps

  • Convert Jupyter Notebooks to Python

  • Configure Jupyter kernels( IPython, IRKernel, IJulia etc) properties

  • Support to execute Notebooks/scripts on Agent

  • Extract graph/map/images from the code

  • Each build step can be associated with a machine learning task

  • Support for Windows and Linux

Future improvements

  • Improving performance of the plugin

  • Try to implement JENKINS-63377

  • Support parameterized definitions in Notebooks JENKINS-63478

  • Increasing testing code coverage

Multiple language kernel support

If there are existing kernels in the system, user will be able to configure in the global configurations in order to apply in the builder/step configuration.

Some popular interactive kernels

  • IPython for python

  • IRKernel for R

  • IJulia for Julia

  • IJavascript for javascript

More kernels and installation guides are found here. https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

Dump images and graphs

Text output will be displayed in the console log. At the same time images/graphs/heat maps and HTMLs will be saved in the workspace. An action is shown in the left panel to display images in realtime. Due to the Content Security Policy of jenkins, some HTMLs which contain harmful javascript may not render in jenkins UI.

action image view

Fixed bugs

There were more bugs identified and fixed with many interactive testings. Setting the working directory of kernels was a big issue while getting datasets/files by script. Zeppelin process launcher was bypassed to fix this issue.

Patch version released

A major bug which was created while setting the process working directory had patched in the v1.0.1. The latest release is more stable now.

Acknowledgement

Machine Learning plugin had been developed under GSoC 2020 program. A huge thanks to Bruno P. Kinoshita, Marky Jackson, Shivay Lamba, Ioannis Moutsatsos and Org admins for this wonderful experience. I would be grateful for contributing this plugin continuously and more in Jenkins.

Resources

About the Author
Loghi Perinpanayagam

Computer Science and Engineering Student at University of Moratuwa, Sri Lanka. He has been selected for Machine Learning plugin for Data Science in GSoC 2020 for Jenkins project. Hightly interested and contributing on open source projects.