Prevent students from changing autograding tests

I’m just getting my feet wet with autograding. Nice feature! It looks like it creates a .github directory, with a workflow that runs the tests.

Perhaps I’m just an evil supervillain at heart for thinking about this, but what prevents a nefarious yet smart student from changing the contents of the .github directory to make all of the tests trivially pass?

6 Likes

Custom actions are the way to go I’d say.

@sakethramanujam: I’m still new to actions, but I’ve started learning. Wouldn’t the code for a custom action still be in the .github directory?

2 Likes

This is coming very soon :slight_smile: Likely in the next month, we’ll do an announcement when it ships.

3 Likes

Sounds good. Thanks, @d12.

1 Like

My idea is to write an action that will pull tests from a private repo and run the tests and return test report.

@sakethramanujam: So… what would stop the student from modifying the action itself in the .github/workflows directory? I.e., can’t the student change the yaml in there so that it ignores the code (or the custom action) that pulls from the private repo, and executes their own action instead?

2 Likes

Ahhh! How did I miss this. Interesting problem to think of :thinking:

A related post: How to anti cheating for student's repo?

GitHub Classroom is planning on solving this by saving the tree sha of the .github/ directory when we setup autograding. Because of how the Git FS works like a Merkle tree, any change to the autograding files will change the tree sha.

On our dashboard, we can show you if this “tamper seal” has been broken. GitHub doesn’t have good controls to prevent someone with write access from changing files in a specific directory, but we can tell if they’ve been changed.

In the future, we’d like to implement a “private tests” feature similar to what @sakethramanujam mentioned. But this is a little ways out right now.

3 Likes

Thanks, @d12. That’s a clever idea.

Looking forward to this new feature. I’ve been using Travis CI. It has a similar flaw: it won’t be long before a clever student figures out that he can use the debug access to decrypt the private tests.

Love this idea ! thx @d12

What would happen if you protect the branch with the action in it and use secrets to encrypt any private test cases? I imagine you’ll still have the perennial autograder problem of the student code being able to dump anything it’s capable of touching, but as long as you don’t give students admin on the repo my understanding is that this would prevent the autograding tests from being changed.

This workflow also helps nudge students toward a “develop on a dev branch and then merge into master” good habit.

Hi Has this been implemented yet? I am exploring GitHub Classroom and my students can and will change the tests if they have access to do so. That undermines my use case to use autograding here. Thanks!

Hi guys,

I find that private test would be an awesome feature to have. In addition it would be great to support randomization in the test suite.

1 Like
© 2017 GitHub, Inc.
with by
GitHub Education