Let's talk autograding!

Hi folks!

We’re working through some potential improvements to the current autograding experience on Classroom. I would love to talk with more of you folks about what is going well or not so well with autograding, or why you’re not using it.

If interested, you can schedule some time using my Calendly link

Looking forward to it :rocket:

2 Likes

I love the auto-grading features when they work! I use them so my students can get instant feedback.

I’m not that skilled or experienced in creating unit tests so it was difficult to get Catch2 working with my C++ assignment templates. Same went for python. I had never written any unit tests because I got my degree in math then a minor in computer science so there wasn’t much covered in terms of testing.

I think it would help people a lot more if there was template and guide on how to copy the template and make your own tests. I know there’s tons of frameworks but if you just start with a few to get the ball rolling it would help.

As a jumping off point I know that you can also make actions, so understanding more of those and how they can interface with other things might be helpful for us so that we could potentially link grades to our LMS.

Or I could be way off… Honestly I’m not much of a developer, but I enjoy teaching computer science!

(I was unable to login using my work account @mabre (502 bad gateway, already contacted support), so using my private account now.)

We have used GitHub Classroom for weekly assignments in our Java beginners course this semester for the first time. Autograding looked promising at first, but we didn’t use it because the following features were missing (or we didn’t find them):

  • We want to be able to set-up the automatic tests and their respective points in our local copies of the git repositories, so that they are easily re-usuable for the next year and we have not to click through an UI for every test case. We had old JUnit test cases and we wanted to recycle them as best as possible. In the end, we set up an own CI pipeline which parsed the JUnit output and updated a points badge which is displayed in the Readme.md in the students’ repositories.
  • TAs must be able to manually modify grades in case students cheated.
  • TAs must be able to manually assign some points (e.g. after checking graphical outputs or free text subtasks).
  • TAs should be able to give text feedback for grades. (We resorted to TAs opening issues with feedback text and the final grades/points, which we then queried via the issues API.)
  • Grades per assignment should be downloadable, preferable via API.
  • Students should have an overview of all their grades.

We want to be able to set-up the automatic tests and their respective points in our local copies of the git repositories, so that they are easily re-usuable for the next year and we have not to click through an UI for every test case. We had old JUnit test cases and we wanted to recycle them as best as possible. In the end, we set up an own CI pipeline which parsed the JUnit output and updated a points badge which is displayed in the Readme.md in the students’ repositories

@gulp21 I also wanted to do this and found a way. It’s not documented though. You can set up the autograder files in a template repo and use this as a starting point for each assignment you create (here’s my template, feel free to use/adapt). Your assignment can then have the code and test config stored in the same repo. Configure the tests using the autograding.json file (specifying the name, run and points for each test, plus whatever else you want to configure). Then when creating new assignments in GH Classroom skip the test UI entirely. This has worked well for me.

I totally understand @Smith-TVDSB’s use case and that there’s a category of users who want GUI-driven test creation for perfectly valid reasons. But please keep in mind that there’s also a category of users who want resuable low-level control over test configuration (e.g. via writing an autograding.json) rather than being restricted by GUI-driven test management.

John,

I just wanted template repositories for specific languages to simplify the learning process. Now that I found some resources I’ve been able to adapt some to specific use cases for assignments. I’d also like to see if I could use hooks with docker to directly interact with a LMS. I would imagine it’s possible to link it to a website but our board might not like us publishing student information on a website instead of an LMS.

Ah yes, sorry @Smith-TVDSB, I misunderstood. I saw “template” and heard “GUI” for some unknown reason. Makes sense.

LMS integration would be super useful.