Overall Specification of a config.json file

You are allowed to have C/C++ style comments in a config.json file. These will be removed before compilation of the autograding executables.

• field: "testcases"
  type: array of testcase objects (defined below)
  REQUIRED

• field: "notebook"
  type: array of notebook objects (notebook docs)

• field: "assignment_message"
  type: string
  default value: ""

• field "max_submission_size"
  type: integer
  default value: 100000

  This is the max size in bytes of all files combined for a website submission.

• field: "grading_parameters"
  type: associative array / mapping from string to integer

  NOTE: These fields are only used as helper checks summing the total of the test case points.

  • field: "AUTO_POINTS"
    type: integer

  • field: "EXTRA_CREDIT_POINTS"
    type: integer

• field: "part_names"
  type: array of strings default value: empty array

• field: "resource_limits"
  type: associative array / mapping from string to integer

  The instructor can override / customize various resource limitations for the testing process. The default values for these resource limits are specified here: [grading/default_config.h][grading/default_config.h].

  Depending on the testcase type, there is a minimum RLIMIT value that will be used, regardless if the limit it set lower in the configuration file.

  If the Testcase is for compilation, it will use the following minimum RLIMIT values:

  • RLIMIT_CPU: 60
  • RLIMIT_FSIZE: 10MB
  • RLIMIT_RSS: 1GB

  If the Testcase is using the submitty_count utility, it will have the following minimum RLIMIT values:

  • RLIMIT_NPROC: 1000
  • RLIMIT_NOFILE: 1000
  • RLIMIT_CPU: 60
  • RLIMIT_AS: RLIM_INFINITY
  • RLIMIT_SIGPENDING: 100

  FIXME: UPDATE & DOCUMENT

  These resources can be for the overall assignment configuration, or per test case.

• field: "allow_system_calls"
  type: array of strings

  The instructor can also override / customize the default restrictions on use of system calls within the student code by whitelisting additional categories of system calls: [grading/system_call_categories.cpp][grading/system_call_categories.cpp] [grading/seccomp_functions.cpp][grading/seccomp_functions.cpp]

  FIXME: UPDATE & DOCUMENT

Specification of a Testcase

• field: "type"
  type: string
  value: "Compilation", "FileCheck", or "Execution"
  default value: "Execution"

  Each test case has a type, the type dictates whether actions are necessary for that test case during the different phases of autograding (compilation, execution, and validation).

• field: "title"
  type: string
  REQUIRED for all testcases

• field: "details"
  type: string
  default value: ""

• field: "points"
  type: integer
  default value: 0

• field: "hidden"
  type: boolean
  default value: false

• field: "extra_credit"
  type: boolean
  default value: false

• field: "filename"
  type: string or array of strings
  REQUIRED for "file_check" and "execution" types

• field: "executable_name"
  type: string or array of strings
  REQUIRED for "compilation" testcases

  FIXME: Should we rename/collapse this field to filename??

• field: "command"
  type: string -or- array of strings
  REQUIRED for "compilation" and "execution" testcases. Not allowed for "file_check" testcases.

  NOTE: simple wildcards allowed.

  NOTE: Each string command should be a LINUX command line that will be executed during either the compilation or execution phases of automated grading.

• field: "resource_limits"
  type: associative array / mapping from string to integer

  _NOTE: See details for assignment level description above.

• field: "validation"
  type: array of validation objects (described below)
  default value: automatic checks added for STDOUT.txt, STDERR.txt, and the execution logfile.

• field: "actions"
  type: array of action strings (described below)
  default value: empty

• field: "textboxes"
  type: array of textbox objects (Described below)
  default value: empty