Usage

Before following this guide, make sure you’ve installed zucchini as described in Installation.

Grading an Assignment

The following section is written as a zucchini workflow that would be used by a TA in a course that already has a zucchini farm set up (as an example, we will use the sample zucchini farm), that an instructor or TA has already prepared the assignment and linked to it on the farm (as an example, we will use the sample zucchini JUnit assignment), and that student submissions are available in directory (as an example, we will use sample submissions on a git repo).

Note that this tutorial expects that you are on either Linux or OSX, that you have access to the terminal (Terminal.app on OSX), that you have installed a Python distribution that’s >=3.4 (we recommend Anaconda for beginners), that you have git installed, that you have JDK 1.8 or higher installed and linked to your path, and that you have gradle installed.

Let’s start by installing zucchini

pip install zucchini

We set up our workspace by entering our identity details:

zucc setup

Then we add the farm for the metadata repository created by our instructor. We name it cs1337-fall1970:

zucc farm add https://github.com/zucchini/sample-farm.git cs1337-fall1970

Then we make a new directory for our grading and change into it.

mkdir zuccsample && cd zuccsample

We list the assignments on our farms to find the one we’re looking for:

zucc list

From the output of this, we find that our assignment is called junit/stacks-queues. We use zucc init to make zucchini pull the assignment configuration into a new directory which will have the assignment’s name. We use our farm’s name as well as the assignment’s name on the farm. Note that detailed information about this assignment, which tests a Stack and Queue implementation using JUnit, can be found on the repository page for the assignment.

zucc init cs1337-fall1970/junit/stacks-queues

Then, we download the sample submissions:

git clone https://github.com/zucchini/sample-assignment-submissions.git

Now we change into our assignment directory, and make zucchini load the submissions we just downloaded. Note that in a real workflow, submissions would likely be loaded through LMS integration modules such as Canvas. Also note that the -d flag for the path loader is used to make zucchini use the directory name (e.g. Alice) as the submitting student’s name as well.

cd stacks-queues
zucc load path -d ../sample-assignment-submissions/Alice
zucc load path -d ../sample-assignment-submissions/Bob
zucc load path -d ../sample-assignment-submissions/Charlie
zucc load path -d ../sample-assignment-submissions/Dave
zucc load path -d ../sample-assignment-submissions/Eve

Then, we start the grading process. This will grade each submission separately and save their results in their folders into the submissions’ meta.json files. Once the grading is done, a text editor will open to show the newly updated grades. Hit :q close it.

zucc grade

Now that we’re done grading, we want to exports the grades our students received. Note that in a real workflow, this would also likely be done through LMS integration modules such as Canvas, which allow for grades to be saved directly onto students’ accounts.

zucc export csv > grades.csv

And we’re done! The grades can be found in the CSV file.

Creating an Assignment

Anatomy of an Assignment

A Zucchini assignment consists of a list of components, each of which itself consists of a list of parts. Like this:

Zucchini aims to streamline the process of converting a student’s submission to a grade in the gradebook, and an assignment instructs Zucchini how to perform this conversion. Indeed, Zucchini downloads submissions, posts grades, and checks due dates for entire assignments, even if they consist of multiple components.

Components represent the smallest pieces of an assignment that Zucchini can grade independently. Usually, this means each independent file in the submission has its own component. Examples of components:

• A test class which tests a particular class in the submission in a JUnit-based grader
• A test suite in a Libcheck-based grader
• A subcircuit in a CircuitSim circuit
• A set of prompts in a prompt grader

Parts represent the smallest result in grading a component that deserves its own weight. We generalized parts because we noticed all of our backends had them. Examples of parts:

• A test method in a JUnit-based grader
• A test in a test suite in a Libcheck-based grader
• A test of a subcircuit in a CircuitSim circuit
• A prompt in a prompt grader

Now, here is a concrete example of the diagram above for a homework with a CircuitSim circuit fsm.sim and a Java file BitVector.java:

Weights

Zucchini weights components and parts relatively. That is, a component \(i\) is worth \(\frac{\text{weight}_i}{\sum_k \text{weight}_k}\) of the grade.

So for the following assignment:

the rubric is actually:

Component	Percent
fsm.sim (One-hot subcircuit)	50%
fsm.sim (Reduced subcircuit)	16.67%
BitVector.java	33.33%

Parts have the same relationship with their parent components. So a part \(j\) of a component \(i\) is worth \(\frac{\text{weight}_i}{\sum_k \text{weight}_k} \times \frac{\text{weight}_j}{\sum_l \text{weight}_l}\) of the grade.

Don’t let the decimal points above mislead you: Zucchini calculates grades with rational numbers internally, so you you don’t need to worry about floating point screwing up or perfect submissions getting a 99.99 or anything like that (lc3grade had this problem).

We added relative weighting because we didn’t enjoy twiddling with weights until they summed to 100. If you do, you can make all the weights add up to 100:

Assignment Configuration

The directory structure for an assignment my_assignment looks like:

my_assignment/
    zucchini.yml
    grading-files/
        some-grader-jar.jar
        some-grader-file.sh
    submissions/
        Sood, Sanjay/
            meta.json
            gradelog.txt
            files/
                fsm.sim
        Lin, Michael/
            meta.json
            gradelog.txt
            files/
                fsm.sim

You need to create only zucchini.yml and optionally grading-files/. Zucchini will generate submissions/. zucchini.yml looks like

name: Homework X # required
author: Michael Lin # required
due-date: 2018-06-24T18:00:00-04:00
canvas:
  course-id: 2607
  assignment-id: 8685
penalties:
- name: LATE
  backend: LatePenalizer
  backend-options:
    penalties:
    - after: 1h
      penalty: 25pts
components: # required
- name: Finite State Machine # required
  weight: 2 # required
  backend: CircuitSimGrader # required
  backend-options:
    grader-jar: hwX-tester.jar
    test-class: FsmTests
  files: [fsm.sim]
  grading-files: [hwX-tester.jar]
  parts: # required
  - {test: clockConnected,  weight: 1}
  - {test: resetConnected,  weight: 1}
  - {test: enableConnected, weight: 1}
  - {test: outputA,         weight: 5}
  - {test: transition,      weight: 10}
- name: Fully reduced
  weight: 1
  backend: CommandGrader
  backend-options:
    command: "java -cp hwX-tester.jar com.ra4king.circuitsim.gui.CircuitSim fsm.sim"
  files: [fsm.sim]
  grading-files: [hwX-tester.jar]
  parts:
  - text: "banned gates?"
    answer-type: bool
    weight: 2
  - text: "number of incorrect SOP expressions"
    answer-type: int
    answer-range: [0, 5]
    weight: 3

You can find a full list of graders at zucchini.graders.

Farms

Before Zucchini, grading for us meant hunting down the grader archive on either Slack, Google Drive, or GitHub. Adding to the confusion, sometimes these different sources would get out of sync, forcing TAs to regrade their section all over again. Zucchini offers a solution to this you’re probably already comfortable with: git.

TODO: Finish