Create a New Gradeable

To create a new gradeable, instructor users should click the “New Gradeable” button in the top right corner, under the Submitty logo. Fill out this form (details in the following sections) and press “Add Gradeable”.

Most of the fields on the Create/Edit Gradeable form can be changed later. You can return to this form from the main page by pressing the “Edit” button at the end of the row for this gradeable. Two fields cannot be changed after the initial gradeable creation:

IMPORTANT NOTE: You should not change the TA Grading rubric after TAs have started grading!

Types of Gradeables

A “Gradeable” is any single item that will be graded automatically and/or manually by the instructor/TAs. We offer 3 basic types of gradeables: electronic submission, numeric, or checkboxes.

Electronic Submission should be used for homework/project/exercise file uploads by students (e.g., code, .pdfs, images, etc.) or submitted via VCS. Electronic Submission can be automatically graded, fully manually graded, or a combination of both.

Numeric is used for data entry of quizzes or exams where the scores for each student are a simple array of one or more numbers (possibly with a short text comment). We provide a spreadsheet like interface for manual data entry of the numeric scores per problem that are summed for a total score. Or you can just enter the total score as a single column. We also provide a .csv upload if your numeric scores for this gradeable are already in an electronic format.

Checkpoints should be used for manually graded items with one or more parts that are marked as full, half, or no credit. This is commonly used for in class exercises (e.g., lab or recitation) where the TAs verify the students have completed the exercise and demonstrated sufficient competency with the material. Similar to numeric gradeables, we provide a spreadsheet-like interface for manual data entry.

Autograding Configuration Path

For Electronic File uploads you must specify the full path to the autograding configuration config.json file stored on the submission server. (More details on the Assignment Configuration page.)

If you do not need any automated testing or grading; that is, if you are collecting files for the TAs to fully manually grade, choose the “no autograding” sample assignment configuration:

/usr/local/submitty/more_autograding_examples/upload_only/config/

Gradeable Dates & Times

From the create/edit a gradeable page you can specify (or change):

Grading User Groups

Grading permission denotes the lowest privileged user that may grade the gradeable. Grading permissions fall into 1 of 4 categories.

Both full access graders and limited access graders are assigned sections of students to grade for each gradeable (by registration or rotating section, see below). When these users log in to the system they are presented with only the grading work they have not yet completed. The difference between these grading types is that full access graders can navigate to inspect or regrade any student for that gradeable (even sections they are not assigned). Limited access graders are restricted.

Grader Assignment Method

Grader Assignment Method can be chosen for each gradeable. It specifies the set of students assigned to each grader.

Build/Debug all Grading Configurations

After adding one or more new electronic submission gradeables, or if you have modified any of the assignment configuration files, the configuration must be built or re-built. If you are using a system-wide sample configuration or if you are using a configuration you uploaded through the website, the build will happen automatically after you create or edit the gradeable.

However, if you have specified a configuration in a private course repository, (see Assignment Configuration page) you must re-run the BUILD_XXXX.sh script.

  1. Log in to the server. Navigate to your top level directory, e.g.

    /var/local/submitty/courses/s17/course_01/
    
  2. Run the script, e.g.:

    ./BUILD_course_01.sh
    

    To re-build only a single gradeable, you can specify that gradeable id as an optional argument. E.g.:

    ./BUILD_course_01.sh hw01
    
  3. Fix any errors in your configurations, and re-run as necessary.

Test your Assignment Configurations

  1. To manually test your course and assignment configurations, follow the student submission instructions

  2. You can confirm that the files were received by checking this directory:

    /var/local/submitty/courses/s17/course_01/submissions/ASSIGNMENT_ID/USER_ID/VERSION
    

    For direct file upload submissions (single file or .zip) this directory will contain all submitted files and any directory structure contained in the zip file. In addition to the submitted files, in that directory you will also find a .submit.timestamp file. This is the time of submission, not the time that automated grading started/finished.

    For submission via SVN, this directory will only contain the .submit.timestamp file. This timestamp will be used to retrieve the state of the repo at the time of submission.

  3. For SVN submissions, you may also inspect an archive of the files/directory structure SVN repo at submission time in this directory:

    /var/local/submitty/courses/s17/course_01/checkout/ASSIGNMENT_ID/USER_ID/VERSION
    
  4. Once automated testing & grading has completed, you can inspect some behind the scenes details of the automated grading by looking in this directory:

    /var/local/submitty/courses/s17/course_01/results/ASSIGNMENT_ID/USER_ID/VERSION
    

    In this directory you will find the results_grade.txt summary of the automated grading results, and STDOUT, STDERR, execution logfiles for each test case. You will also find log files for the compilation, runner, and validation components of automated grading that are helpful in debugging assignment configurations.

Using Git

Submitty supports utilizing Git for handling submissions by students. When a student hits “submit”, Submitty performs a clone of their repository. Submitty then does a checkout of the last revision made before the timestamp of when they hit the submit button.

To configure Git for an assignment, you will want to select type “Electronic File” and then “Version Control System (VCS) Repository”. You will then be presented with a textbox to type in the repo url to use. If the string does not start with a / nor has a “://” in it, it will combine the contents of the textbox with the base url. You will see the URL/path that will be shown to the users underneath the text box.

Additionally, the text field supports the following list of “variables” which get replaced during execution of Submitty:

Submitty comes with a setup for an internal Git server out of the box, but if you wish to use an external source, such as Github, you may need to configure Submitty to be able to access those repos via SSH. See System Administrator/Setting Up Git for more details.