Autograding using up all Action minutes, timeout doesn't work

GitHub Classroom
#1

I’m using GitHub Classrooms to manage an assignment. One of our test cases causes an infinite loop, but we configured the Test settings to have a timeout value of 3 minutes using the Classroom interface. However, it appears that this timeout isn’t actually enforced by the Action runner – my action ran for 6 hours before being terminated by the system. :frowning:

Are Timeouts supposed to actually stop the program? My guess based on the observed behavior is that either the timeout setting is broken and does nothing, or it only affects the “points” that a student gets on a test, without preventing the action from running longer. If the latter is the case, this really needs to be made more clear in the interface!

Since the test case was automatically run three times when I created a test version of the repo I have already used up all 2000 minutes of Actions allocated to me for the month… I haven’t even assigned the project to students yet!

I saw a similar post linked below and I know I can upgrade to Teams to get 1000 more minutes, but if the actions are left to run for hours on end this isn’t going to help me. How can we enforce a maximum run time for auto grader actions? I strongly suggest GitHub by default enforce a low timeout here since it isn’t hard for a beginner programmer to make an infinite loop!

2 Likes
#2

You can use linux timeout command in the run section of the testcase when you create the testcases. It will also enable you to enforce timeouts in seconds.

2 Likes
#3

@denduuyum - good idea! We’ve been modifying the action yml files to force a timeout, but your solution is a lot simpler.

(I still think that Classroom should make the Timeout field when defining a test actually enforce a timeout to avoid this confusion)

2 Likes