GSOC 2019 REPORT: Continuous Integration Testing (Automation) for Submitty

The aim of this project is to expand the code coverage of the unit and integration tests for Submitty codebase.

Background and Requirements

From the beginning we observed a couple of problems and chanllenges including :

  1. Travis had become to very suitable for the complex Submitty structure. Which holds code in multiple languages
  2. The times taken to complete builds are/were quite significant for travis.
  3. Submitty code needs more and more tests, i.e we have to increase the code coverage and enforce practices to maintain its growth.
  4. We should be able to get a global view of what the system code coverage, in essence having a single coverage report for all languages.

To achieve this feet, I and my project mentors Matt Peveller and Barb Cutler decided on the following work that had to done.

Since this work is just about architecture we also discussed extensively possibilities of completely having Submitty run on docker and not just the ubuntu VMs which take a lot of time to set up and to do tests when working with.


As a result, we expected Buildbot, the highly configurable Continues Integration and Automation tool which we could maneuver to get certain not so common task done for us. This include:


1. Buildbot

In general, I was able to set up and have Builbot run builds on pull requests and also changes to the master branch.

The major things are achieved here are :

*A Submitty pull request showing 4 newly introduced builders powered by buildbot*


2. Submitty PRs

To complement work being done on Buildbot, and also to help me understand the structure of Submitty I had to work on a couple of issues. Most of the work, was related to modifying the python script to improve some aspect of automation.

References, major tasks

Amongts a couple of other issues, some of which are yet to be completed.

Future Work

Following various obstacles faced, during the period of GSoC and various areas spotted for improvement there is more work to be done.

For builbot, we agreed to document the work that needs to be done to improve it’s functionality based on constraints from my end and limitations of Buildbot itself via the use of open issues.


Buildbot Future Work: Issues tagged [future-work]

Thanks Submitty!

Working with and on Submitty, has been a great experience for me. I was greatly motivated by the “killer” sysadmin and problem solving abilities of my mentor Matt Peveller and the highly friendly, supportive and flexible environment created by Barb Cutler our GSoC admin.

Many thanks to Shail Patel, all the GSoC participants and everyone at Submitty.