We utilize the Travis CI platform to enable continuous integration on all commits and pull requests against the GitHub repository. As part of this CI, we validate that the code passes the linters / static analysis, the python and PHP unit tests, the Autograding Integration Tests, and the Site End-to-End Tests.
To help keep things organized, we utilize a concept of build stages, which we use to logically group test suites, and structure the stages roughly from fastest to slowest. This allows us to fail relatively quickly in Travis-CI if for example your code does not pass the linter / static analysis without having to spend all the time setting up all of Submitty. The build stages that are defined are:
- Set-Up Cache - Makes sure all pip packages are installed, speeds up subsequent pip install usages
- Run Linters / Static Analysis
- Run Unit Tests (Python and PHP)
- Run Site End-to-End Tests
- Run Autograding Integration Tests
To see how Travis-CI is configured, see the .travis.yml configuration file.
To help keep our configuration from growing too unwieldy, and not require a lot of duplication for the stages, we utilize a concept of YAML Anchors.
Unfortunately, Travis-CI does have some flakiness in running the End-to-End tests. If you notice a test fails, especially for a given reason of ` (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)`, it is suggested that you restart that particular tests and any subsequent tests in later build stages. To do that, go to Travis-CI/Submitty, log-in, and then you should see a button in the upper right corner of the job to restart the entire job, as well as a button at the end of each row to restart that particular row:
Finally, for really spurious failures, it may be useful to launch a debug build, which will allow you to SSH into the machine and directly interact with it to see what went wrong and why it’s failing.
- The End-to-End tests keep failing on Travis, but they work locally!
Something to remember is that each build of Travis starts from a completely blank slate, and runs through the regular installation process to get up and running. Often times, these sorts of failures are a result of only testing the code / upgrade process for an existing Submitty installation, and not the path of a new Submitty installation.