Keep tests for autograder secret

GitHub Classroom
#1

What are the best ways to keep “secret” tests secret when using autograding with GitHub Classroom?
In other words, what do you all do to keep students from being able to see the tests that are run by the autograder workflow?

I have two main areas of concern: (1) I don’t want the students to see the source code for the tests, and (2) I don’t want the students to be able to ascertain the test inputs by putting print statements in their code. Ideally, (3) students wouldn’t be able to run my tests locally; but, that is a lesser concern at the moment.

My primary concern at the moment is Java. But, in the fall, I’ll also need a solution for C.

I know that there are some new features coming. I’m just wondering what techniques you all have developed in the meantime.

3 Likes
#2

I don’t keep the tests in secret to my students. I explained to them where tests are, how to run and encourage them to write their own tests.

Until now, I don’t have to worry about it. As we know, not all students do the homework fairly, so I got some repos changing the tests, but are really few. In the revision, I do the comments, so, even the test pass, it will not be graded.

#3

I had similar concerns when I first started experimenting with the auto grader. I teach a second-year (high school) programming course in Java and want students to benefit from automated tests so they can confirm whether their submitted code is (generally) performing as intended. I was worried that students might look to the tests and “short circuit” thinking through corner cases; however, on further reflection, I now believe:

  • Students benefit from seeing and thinking through well written test cases
  • My class load is low enough that I still manage to get “eyes on” student work when reviewing pull requests
  • The types of projects that I’m using Github Education for (mainly, exercises) are just one tool for determining student proficiency – I have other, complementary, assessment strategies including open ended projects, instructor monitored pair programming, live coding and code walkthroughs

That said, if I were to use Github Education for assessment or for coding competitions, I would need to keep the test cases hidden and would also appreciate hearing from others who may have found a way to do this.

#4

I also think that showing the students seeing the tests is a good policy.
I like it so much that me and colleages write the Tests Using doctests
(Tests that are written as part of the documention of the function)
That way the student gets the following advantages:

  • The can see example of input and expected outputs from their exercises.
  • Hopefully by imitation they learn to write the tests first and then the function. I.e. they learn Test Driven Development.
  • Also by imitation teach them to write good documentation for their functions.
#5

You are correct: students do benefit from seeing well-written tests. But, at some point, they need to begin to think up and write their own tests. In our first semester programming course, we provide almost all the tests to the students. During the second semester, we begin providing fewer tests.

I try to get students to follow a TDD-like workflow: I give them a few tests to get started, then ask them to write their own tests based on the project spec. After they have thought about and written the tests then they write their code and run it against my “hidden” tests. If their code passes their tests but fails my tests, then they know that they need to go back and figure out what tests they forgot to write. (I, of course, provide hints as necessary.)

The whole point of “hiding” tests is to force students to practice coming up with their own tests so that when they are out of school and on their own they don’t release buggy code because they overlooked test cases. As with coding, good testing skills take years to develop. You don’t become a good tester from a few lectures in a software engineering class. If we want our graduates to test well, we need to make them practice as often as is reasonable.

1 Like
© 2017 GitHub, Inc.
with by
GitHub Education