Python Unit Tests

The following Python modules of Submitty are unit tested:

Migrator (under migrations/ folder)
Autograding (under autograding/ folder)
python_submitty_utils (under python_submitty_utils/ folder)
submitty_jobs (under sbin/submitty_daemon_jobs/ folder)

Running Python Unit Tests

To locally run the Python unit tets, you need to cd into their directory and then use the unittest module to run them. To run all the tests for a particular module:

python3 -m unittest discover

To gather code coverage of the unit tests, run:

coverage -m unittest discover

For example, to test the Migrator module:

cd migrator
python3 -m unittest discover 
# or
coverage -m unittest discover

To test a portion of a module, pass in the module name (filename), class name, and then function name depending on how specific you wish to go. For example, for the migrator:

# run whole test module
python3 -m unittest tests.test_cli
# run one test function
python3 -m unittest tests.test_cli.TestCli.test_no_args 

Submitty Daemon Jobs

Make sure the python system dependencies are locally installed, this can be done by running the following command at the root of the project structure.

python3 -m pip install -r .setup/pip/system_requirements.txt

For the submitty_daemon_jobs unit tests, you will need to set the top level directory to sbin/submitty_daemon_jobs when running them. This can be done by the following command

#in the sbin/submitty_daemon_jobs directory

python3 -m unittest discover tests -t .

Or if you’re using coverage

#in the sbin/submitty_daemon_jobs directory

coverage run -m unittest discover tests -t .

Writing Python Unit Tests

The unittest documentation includes good examples and general information for writing tests.

Getting started, the structure of the unit tests is such that the python module lives under a source directory, and the tests are under the tests/ directory. There should be a one-to-one correspondence between test file and source file. For example, for migrator, there’s migrator/cli.py and tests/test_cli.py. For particularly complex modules, it may be beneficial to break up the tests for that module into separate files focusing on one aspect. However, this is an exception to the norm, and for most things, the one-to-one design works well.

For actually writing a test, we start with a basic example:

import unittest

class TestStringMethods(unittest.TestCase):
    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

Here, we see that to create a unittest module, we import it, create a class that extends unittest.TestCase and then every method under that class is prefixed with test_. With each function, there is a series of assert methods that can be used to verify behavior.

Python Unit Test Fixtures

While most unit tests can be largely stateless, there may be times you wish to have common initialization for a particular test module. This is called a “test fixture”, which involve some preparation for running a test, and then the cleanup afterwards. An example of where this might be useful is if each test generates a file, you could have a test fixture to create a new random directory for that file to go into, and then a test fixture to delete that folder after the test runs. unittest uses the functions setUp and tearDown to represent that:

import os
import shutil
import unittest

class TestModule(unittest.TestCase):
    def setUp(self):
        self.directory = os.makedirs('test')
    
    def tearDown(self):
        shutil.rmtree('test')

For more details, see the section on Organizing Tests from the unittest documentation.

Submitty