Keep tests for autograder secret

GitHub Classroom
#1

What are the best ways to keep “secret” tests secret when using autograding with GitHub Classroom?
In other words, what do you all do to keep students from being able to see the tests that are run by the autograder workflow?

I have two main areas of concern: (1) I don’t want the students to see the source code for the tests, and (2) I don’t want the students to be able to ascertain the test inputs by putting print statements in their code. Ideally, (3) students wouldn’t be able to run my tests locally; but, that is a lesser concern at the moment.

My primary concern at the moment is Java. But, in the fall, I’ll also need a solution for C.

I know that there are some new features coming. I’m just wondering what techniques you all have developed in the meantime.

7 Likes
#2

I don’t keep the tests in secret to my students. I explained to them where tests are, how to run and encourage them to write their own tests.

Until now, I don’t have to worry about it. As we know, not all students do the homework fairly, so I got some repos changing the tests, but are really few. In the revision, I do the comments, so, even the test pass, it will not be graded.

2 Likes
#3

I had similar concerns when I first started experimenting with the auto grader. I teach a second-year (high school) programming course in Java and want students to benefit from automated tests so they can confirm whether their submitted code is (generally) performing as intended. I was worried that students might look to the tests and “short circuit” thinking through corner cases; however, on further reflection, I now believe:

  • Students benefit from seeing and thinking through well written test cases
  • My class load is low enough that I still manage to get “eyes on” student work when reviewing pull requests
  • The types of projects that I’m using Github Education for (mainly, exercises) are just one tool for determining student proficiency – I have other, complementary, assessment strategies including open ended projects, instructor monitored pair programming, live coding and code walkthroughs

That said, if I were to use Github Education for assessment or for coding competitions, I would need to keep the test cases hidden and would also appreciate hearing from others who may have found a way to do this.

#4

I also think that showing the students seeing the tests is a good policy.
I like it so much that me and colleages write the Tests Using doctests
(Tests that are written as part of the documention of the function)
That way the student gets the following advantages:

  • The can see example of input and expected outputs from their exercises.
  • Hopefully by imitation they learn to write the tests first and then the function. I.e. they learn Test Driven Development.
  • Also by imitation teach them to write good documentation for their functions.
#5

You are correct: students do benefit from seeing well-written tests. But, at some point, they need to begin to think up and write their own tests. In our first semester programming course, we provide almost all the tests to the students. During the second semester, we begin providing fewer tests.

I try to get students to follow a TDD-like workflow: I give them a few tests to get started, then ask them to write their own tests based on the project spec. After they have thought about and written the tests then they write their code and run it against my “hidden” tests. If their code passes their tests but fails my tests, then they know that they need to go back and figure out what tests they forgot to write. (I, of course, provide hints as necessary.)

The whole point of “hiding” tests is to force students to practice coming up with their own tests so that when they are out of school and on their own they don’t release buggy code because they overlooked test cases. As with coding, good testing skills take years to develop. You don’t become a good tester from a few lectures in a software engineering class. If we want our graduates to test well, we need to make them practice as often as is reasonable.

4 Likes
#6

Did you find a solution for hiding your tests? In my case I am most concerned with students manipulating the test code.

#7

Unfortunately, No.

1 Like
#8

Yes, it would be great if there was a folder in the template homework repository that would not get cloned into the student version of the assignment. This would allow testing without exposing the tests.

In my case, testing based on output is tricky because of multi-string output which is sensitive to indentations and character encodings used.

#9

I will be teaching for the first time with github classroom this semester and have already built out the assignments with unit tests. It was surprising when I saw this myself… it would be nice if there was some method.

I think modifying the unit tests to cheat is a bit of a risky prospect on the student’s side since these changes will be quite obvious, but since I am relying on unit tests entirely for the practice problems I won’t be opening their repos to look for this.

I would imagine that using a tool to minify or obfuscate the unit test code might work for all but the most determined… that is what I would suggest for instructors concerned about this.

1 Like
#10

I think that the following should work, but requires some setup that may be complex for folks. I’m going to be exploring this in more detail later this semester since we may be able to use github actions to replace our current auto-grader infrastructure (INGInious’ documentation — INGInious 0.6.dev0 documentation) so I would be interested in feedback.

The github classroom autograder uses a github action to run the tests. You can set up your own “self-hosted runner” that connects to your github classroom organization (see About self-hosted runners - GitHub Docs).

You can then direct student actions to use your self-hosted runner.

The self-hosted runner should be based on the github runner code ( GitHub - actions/runner: The Runner for GitHub Actions ) but can include additional code. Some of that code could be your “hidden tests” – however, this would require you either putting the tests in e.g. a docker image or on a host you’re using as the runner. I also don’t think that the hidden test cases would be properly “hidden” from the student programs – if they can root around the file system of your runner, they can find it. I haven’t spent time analyzing the github runner code yet.

In the INGNious system we currently use ( https://inginious.readthedocs.io ) the students code is run in a separate container than the grading script / program and there are containers ( GitHub - UCL-INGI/INGInious-containers: Additional containers for INGInious. INGInious is a free and open-source platform for secure and automated code assessment. ) that then provide support routines to provide meaningful feedback to students and to facilitate running tests. Overall, it’s a nice system and from an educational perspective better than Github actions because it can integrate with Canvas/Moodle/etc and shove grades in the gradebook. However, it’s not currently integrated with github actions / etc.

#11

You could use a private repo to store the tests in. Then modify the classroom.yml file to checkout the private repo into the student’s repo before autograding is run.

You might need to generate a personal access token and add it as a secret in your classroom’s organisation.

classroom.yml will then look something like this (I haven’t tested this):

    on: 
      push:
        branches:

    jobs:
      build:
        name: Autograding
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v2

          # checkout secret tests
          # my_pat is the variable name used when adding the secret to org
          - name: Check out tests private repo
            uses: actions/checkout@master
            with:
              repository: classroom-orgname/test-repo-name
              token: ${{ secrets.my_pat }}
              path: tests

          # run autograding tests
          - uses: education/autograding@v1

See possible to checkout or clone another repo? · Issue #24 · actions/checkout · GitHub

The private repo will get checked out into the ‘tests’ dir. You’ll need to point your test commands to whatever tests you have under that dir.

Students won’t have access to the tests repo, but enterprising students could modify classroom.yml. For example, add -run: cat tests/* tests/*/* to the end, then read the output from the action logs. So it’s not foolproof.

#12

I was not able to make this work :frowning: Mainly because Github is unable to change the workflow once it has been set for an assignment

#13

To improve on Mark’s suggestion:

  • Put the path as a secret, which makes them work a little harder to understand where the tests are
  • Obfuscate by using a less explicit build name and test-repo-name
© 2017 GitHub, Inc.
with by
GitHub Education