To verify expected behavior of Submitty, we use various test suites, broken up per component. Currently, much of the testing is quite sparse, and we encourage all developers to add relevant tests with their PRs. Much as Submitty is made up of many composite pieces, each piece has a different test suite. However, these tests can be roughly broken down into the following high-level types:
- Static Analysis / Linting
- Python Unit Tests
- PHP Unit Tests
- Site End-to-End (E2E) tests
- Autograding Integration Tests
- Rainbow Grades Functional Tests
To illustrate the difference between unit testing, integration testing, and end-to-end tests (also often called functional tests) and what they are for, we recommend the following articles:
These tests are run as part of a Continuous Integration process on GitHub actions. Each of the tests can also be run on your local computer – developers are encouraged to run and debug the relevant tests on their own computer. Please see the page on it for more discussion on how it functions.
Initial Set Up
We encourage people to run most of the tests using their local machine, not within the Vagrant VM. This is especially true of the E2E tests which require a web browser, and for debugging purposes, is useful to not run it headless. To set it up, you’ll need to install a handful of dependencies:
- PHP (and some extensions)
Installing these should be straight forward on macOS (using Homebrew) and Linux distros using its package manager. We provide the instructions for Ubuntu, and we expect users of other distros should be able to adapt it to their needs.
Windows users are encouraged to install Windows Subsystem for Linux (WSL) and then follow the Ubuntu instructions. However, for E2E tests it is better to run them directly on Windows. Make sure that you have Python installed to be able to install dependencies using pip.
brew install python php composer pecl install xdebug
sudo apt-get install php-cli php-mbstring php-xml php-xdebug php-curl php-zip php-sqlite3 composer sudo apt-get install python3 python3-pip
Now, to install dependencies for Python on either OS:
pip3 install PyYAML pip3 install python-dateutil pip3 install tzlocal pip3 install "sqlalchemy<1.4.0" pip3 install jsonschema pip3 install jsonref pip3 install docker pip3 install paramiko pip3 install psycopg2-binary pip3 install coverage pip3 install --pre selenium pip3 install flake8 pip3 install websocket pip3 install websocket-client
To install the PHP and JS developer dependencies:
cd site composer install npm install
Note: Normally one can run Composer installed packages by direct execution,
vendor/bin/phpunit. However, due to the integration of
Submitty installer and Vagrant, packages in the vendor directory will
lose their execution bit, so you will need to specify
php before the
thing you are attempting to execute (e.g.
Please use the sidebar menu for additional documentation for each type of test.