Blog post proposal: Automated grading via Travis CI

(Omar Shaikh) #1

Hi fellow Educators!

I teach a C++ and Operating Systems course at San Francisco State University. I am really excited about the wonderful contribution that GitHub has towards education. I was able to completely auto-grade C++ assignments using GitHub+ GTest+ TravisCI. I met with TravisCI at GitHub Universe. If you aren’t familiar with TravisCI, they have a very easy to use CI system and also provide education packages for organizations. I can contribute towards a blog post that would outline the start to finish workflow that instructors can use to easily setup their auto-graded classroom.

@sanicki Any thoughts on how we could build out this blog post? Ideas from the community are welcome!


Private Test Suites
(John Britton) #2

@Omarasifshaikh This is great to hear. If you want to start working on a post and share it here we’d be happy to give feedback and try to get it published more broadly.

(Richard) #3

I use travis CI as part of assessment, although not automated and only as part of the assessment. In a nutshell, for java applications, I provide some complete cucumber feature and some empty cucumber feature files (no scenarios). I ask students to make the existing tests pass and then add scenarios, then make them pass.

I assess this by viewing the output of their travis builds, but manually, as I also assess the quality of their tests, and other aspects of their code and design.

Much much easier that download zips of their code and trying to get them to run.

If this fits with the topic, I’m happy to contribute

(Omar Shaikh) #4

@rcraggs yes that sounds interesting. I decided to build the workflow for the same reason that it was very inefficient for graders to try to compile and then grade HW assignments. Allowing the students to test their solutions before submission enables them to iterate. The tests have to pass, so debugging skills are also required. Overall, it provides a much better learning environment

(Omar Shaikh) #5

Thanks @johndbritton , Ill start working on it.

(Diego Ocampo) #6

@Omarasifshaikh that sounds great and useful. Are you using such configuration with private repositories?

(Omar Shaikh) #7

@diegoocampoh , Yes, every HW assignment is in its own private individual repository. I use teachers_pet to automate that process. It will be explained in detail in the blog post draft, feedback on that would be very welcome! :smiley:

(Josh Orndorff) #8

Sounds wonderful. I’ve just started using Travis CI after a few years of teaching git with github. Sounds like you’ve had good luck with teachers_pet. I’ve been using github classroom and wonder about the workflow there. After reading your workflow with teachers_pet I can contribute my findings about github classroom. Did I miss the link to the blog post?

(Sungchul Choi) #9

Amazing~ I am also using Unit Test based autograding system for my Python Class. Recently, I am leading Web Application Development clsss with Django, but the autograding system that I developed could not be used anymore :(. I’m really expecting your post and, If I have a chance, I want to contribute the project.

(Adrien Joly) #10

Awesome! :smiley:

I created my own web app for individual student entry and auto-grading (, and I would really like to automate my process further, so I’d love to read from you guys and give feedback / help if necessary!


(Sam Black) #11

I’d love to help out and get something like this working with JUnit and Java.

(Thomas J Bradley) #12

I don’t teach Java, so this isn’t exactly what you’re looking for—I teach HTML, CSS & Javascript.

I tried using TravisCI for a while doing confirmation of assignment requirements. But for me, the process was too slow. I had to do, probably more complex (and time consuming) tests, like running the HTML & CSS validators, not just standard unit tests. The problem we ran into was time: TravisCI became too slow running the tests. Students would wait 20 minutes or more for the tests to complete and if there was a deadline it ended up being too stressful. Especially if there are 25 students pushing causing TravisCI to run all at once. It would get super bogged down.

Because of the stress of the students and the frustration, I abandoned TravisCI and built a desktop app with Electron. The app runs whack-load of tests against their code locally, needing very little network access. So it’s super fast, and really more customizable that what I got from TravisCI.

(Matt Price) #13

@thomasjbradley , I’m really interested in this project. I teach a Digital Humanities class where the coding requirements are pretty simple, and my own coding skills are not fantastic. I would really like to start giving students unit tests for their code – which I would like to make pass/fail – so that they can be sure the code runs before handing it in. Can you share your work here? Thank you so much!

(Lindsay Ward) #14

I’m keen to see anything you’ve done on this. It would be a great extension of GitHub Classroom if we could see some automated testing with CI.


(Vanessa) #15

More to come from @Omarasifshaikh’s awesome work soon @lindsaymarkward. Keep your eyes peeled to

(Lindsay Ward) #16

Thank you. Please announce it here as well. I’d rather get a notification than keeping my eyes peeled… that sounds painful :wink:

(Ricardo Martin Marcucci) #17

Thanks, we are planning to do some automating grading for exercises, so this post will come in handy

(Vanessa) #18

Will definitely post here :slight_smile:

(Vanessa) #19

@rmarku and @lindsaymarkward this post is alive!

(Omar Shaikh) #20

Thank you Vanessa and GitHub for facilitating the post. I can also answer questions here if anyone had any.