Yes, they run all the tests all the time. I’m still figuring out what the best setup is. This is for a class using the textbook & modified versions of the labs from “Computer Systems: A Programmer’s Perspective”. Since I already had well-developed testing & autograding code, I wanted to exploit what I had while also using a single “golden reference” environment to avoid platform dependent issues. In addition, some labs have a “performance score” and I wanted everyone evaluated on the the same hardware.
For some labs, I break out the individual tests a single workflow & actions with multiple ‘run’ statements and the students see each individual test passing or not.
That strategy assumes the tests build on one another – i.e. that you’d have to pass test1 in order to pass test 2, etc.
Another strategy (again, still figuring out what makes the most sense…) is to e.g. 2 workflows with multiple actions. For example, in this “attacklab” (buffer-overflow and ROP) assignment, there are different tasks the student is to do (various forms of code-injection, etc).
There are two “workflows”. The first named “grade-first” is the “checkpoint” that is due by a specific date. The second is is “grade-rest” which needs to be finished by the due-date. In this case, I used two workflows (grade-first, grade-rest) and each had multiple jobs (e.g. run-ctarget-l3, run-rtarget-l2, run-rtarget-l3). The jobs can be run in parallel by default ( Workflow syntax for GitHub Actions - GitHub Docs )
For simplicity, the grading script looks through the list of workflows and keys on the label – i.e. a deadline is expressed for each workflow name:
I only used artifacts to record a grade for one such lab (shown above) since the others all use an oral-grading interview.
The things I need to figure out is the difference between actions and the “check_run” interface, how to get the correct artifact for a specific commit (I just use them in time-order) and options such as “best” vs. “most recent” grade, etc.
For another class, we’re going to combine Github Classroom with the INGInious autograder ( INGInious’ documentation — INGInious 0.6 documentation ). Students will still develop assignments using Github, but the final grading (as opposed to incremental checks) will be done using INGInious. There are some benefits including direct score reporting to the LMS and it’s also slightly more secure since the grading script does not need to be contained in the student repository - this allows us to replace portions of the students assignment with standard elements (pulled from a repo). While I can do that using Github Actions, it means that we either need to pull from a public repo or we expose a PST using a secret – while it looks like Github takes steps to mask or hide the PST it’s not really designed for an adversarial attack.