Submission and Grading Workflow (Redux)


(Matt Price) #1

Re-reading through @mozzadrella’s recent post on assigning and submitting after some sleep, I think I am finally starting to see how review of student work in GH Classroom is supposed to work:

you can have [students] commit directly to their master branch, and then you can click into each individual commit to give feedback in-line.

I can see how this integrates well with the existing GH tools. It’s not ideal for my students, who are new to git. The git history on their repos can be really quite funny, with lots of flailing around. It would be hard for me to comment on individual commits that way. What I really want is a way to comment on their final code, the way I would comment on a PR.

There’s a detailed discussion of various other options in this post from last summer. It sounds like per-commit commenting is the most commonly-used method, though when I first read through the thread I thought that PR’s were being more widely used. However, I think the PR-comment function is mostly used by professors outside of Classroom. It’s the method I used in my first assignment this term, which uses “vanilla” github rather than the classroom mechanisms.

So my first question was, is there a way for me to achieve this (comment on the final product rather than individual commits) in the current classroom setup? At first it seems like the answer is no, since (I believe) there’s no way to submit a PR between 2 repositories owned by the same organization, and there isn’t a “comment directly on code” feature in GH.

This issue is a feature request for full-code commenting. It also links to a blog post which suggests creating a new empty branch (say, “teacher-comments”) and then creating a new PR to merge student work into that branch. That sounds pretty good – though maybe one would want to start the branch at the last commit in the starter code, so you really only see the student code. Still, in a large class it would be difficult to keep track of everyone’s work, since the PR’s are not centrally listed anywhere.

Discussion in @iluvatar1’s post from last summer about gathering submissions has some interesting contributions about scripting various workflows.

Thinking about all of these things together, I wonder if an ideal submission & grading process would look something like this:

  • create an assignment in Classroom as usual
  • at the deadline, create a “submisison” tag or, I guess, branch, in every student repo as described by @sfogarty in this reply to @iluvator. I’m not sure what such a script would look like, but maybe it would use one of the command-line github clients. The script needs access to your GH credentials, though, since these are private repos.
  • create new “teacher-comments” branches in each repo, starting from the last commit in the starter-code
  • initiate PR’s from the “submission” branches to the “teacher-comments” branches
  • collect all the PR URL’s in some format, say CSV or JSON, that allows association between the PR and a student record (this seems essential to me)

Then individual teachers could integrate the PR URL into whatever grading mechanism we normally use. In my case, it’s a twisted ad-hoc undocumented un-updated workflow for Emacs and org-mode, but you could presumably use some more rational method :slight_smile: The only thing that seems to me to be missing here is a way to update assignments after the inital sharing (I asked about this last night in another post).

I would love to hear (a) what other people think of this proposed workflow and (b) some suggestions for implementing its parts. Most of it I guess could be done on a local laptop with a shell script, but it might be more elegant to use the github API instead, especially since presumably you’d need the API to initiate the PR anyway. In either case I’m not sure how best to handle authentication and repo discovery.

Would really love to hear thoughts from the community!


Assigning and Submitting with GitHub Classroom
Updating assignments after creation?
How to review HTML files?
(Andreas Mueller) #2

From my understanding, there is no way to comment on the final submission.

I haven’t tried it yet, but I think the correspondence between submission and students is now handled via the unique identifiers in the classroom. Or I think that’s what it’s supposed to do at lesat.

Last time I checked, updating the starter is not possible (you could encourage people to pull from an updated repo?). Updating the starter doesn’t seem to fit well within the git framework.

From my point of view, using the github interface for grading is fundamentally ill-suited and having something like gradescope would make much more sense. Pull request reviews are not optimized for reviewing the same thing 200 times.

I think my workflow will be pulling all the submissions and then grading locally, or trying to use something like https://github.com/Submitty/Submitty (which I haven’t tried yet).


Grading interface?
(Miren Berasategi) #3

Just for the record, this has been my workflow during the fall semester. Students committed to the same repo throughout the semester, as assignments were incremental.

  1. Students were requested to create a tag for each assignment before the deadline and submit that url to our Moodle platform.
  2. I would inline-comment the code for each assignment in the corresponding commit, using ‘blame’.
  3. I would create an issue for each assignment in each student repo with general commentary on the assignment AND the commit numbers with inline comments from the previous step.
  4. I would post the URL of that issue to our Moodle platform as assignment feedback.

It’s far from perfect but it’s been good enough to get us going.


(Ryan Hahn) #4

I have the students create a new branch in their projects called “my_solution” or something similar. Then when they are ready to submit I have them create a pull request. I can then see what files were changed and what they did to each one. Works fairly well but I would love to have the ability to compare a file to another repo. that way I could setup a solution branch and compare the two.


(Romain Komorn) #5

My org could use pull requests, but it’s an extra step that, IMO, isn’t particularly worth the time.

A pull requests means having to create a branch that isn’t master, pushing to that branch, creating the PR, then having graders comment on the PR interface. If a student unintentionally pushes to master, they’ll have a harder time creating their PR.

Even if master is a protected branch in this scenario, I still don’t really like the idea because it leaves the repo with an unmerged branch (unless the grading process accepts the pull requests and merges it). An assignment that’s turned in should be pretty final, IMO.

I feel like this is a lot of “working around” when the solution that seems the most straightforward (to me, anyway) is for graders to be able to comment on the repo in a UI that would be strikingly similar to the current PR view’s commenting ability.

I know that there would still be concerns about “what happens if a repo is pushed to after comments have been submitted”, but I feel like that’s already answered by the current PR view’s way of handling comments pointing to lines of code in prior commits in the branch that have since been overwritten.


(Matt Price) #6

I never got back to this thread after my semester ended in March, but… I agree with @rkomorn that a UI like this would be the best approach, mostly because the burden of branching is an extra cognitive load for students who are usually already struggling with the underlying programming concepts. But the GH Education team would unfortunately have to build the UI first. @mozzadrella do you see any prospect for something like that every happening?