Continuous integration for each submission


(Andreas Mueller) #1

My homeworks include working with continuous integration, as this is a cornerstone of standard software development practices.
So I want the students to set up travisCI (or any other service) for their assignment.
However, since their repositories belong to the organization, it doesn’t show up in their travis-ci feed.
I could make them all members of the organization, but that wouldn’t resolve the real problem:
While travis CI allows for free CI for educational purposes, it provides a single concurrent run. If the repositories were owned by the students, the students would each get one concurrent run.
However, the repository is owned by the organization, which means that the whole organization gets one concurrent run - which is clearly not sufficient for a class of 100 students.

Is there any solution for this issue? Is there a way I can have the students run CI on the repository within the organization?

Also, is there an efficient method to make all of the students a member of the organization?

Thanks,
Andy


(Carol S.) #2

Hi Andreas,

I’m the partnership manager here on the Education team at GitHub. I’ve reached out to TravisCI about this question. I’m going to talk it over with them and will let you know as soon as I have more info. Stay tuned!

Cheers,
Carol


(Andreas Mueller) #3

Great, thank you! I appreciate your help.


(Andreas Mueller) #4

There’s a related issue: I’m trying to make the students use readthedocs. Again, the .org interface doesn’t work because of private repositories, and the .com doesn’t give them access because they are not members of the organization. So I guess making them members will solve that. It would be great if there was an easy way to do that and ensure they can’t read each others repositories.


(Scott Sanicki) #5

Hi Andy,

Would you elaborate on the readthedocs issue in a feature request on the Classroom repository here:

Thanks,
Scott


(Carol S.) #6

@amueller Just wanted to give you a quick update that I’m in touch with Travis about this and we’ll be meeting to talk about it soon. I’ll update you soon!


(Leszek Grzanka) #7

I’d also appreciate possibility to use travis for CI together with github education machinery.
My course starts beginning of March this year. Do you think it is possible to get something ready by that date ?

Best regards,
Leszek


(Carol S.) #8

Hi everyone, I’m meeting with Travis about this today and I will let you know what I hear then!

Cheers,
Carol


(Lindsay Ward) #9

Hi @fossygirl - can you let us know what the result of your meeting was?
I think this integration would be awesome!


(Mathias Meyer) #10

Hello, my name’s Mathias and I work at Travis CI.

While we can’t, at least not at the moment, provide an automated feature for supporting classroom organisations on GitHub (this is due to how we currently integrate with GitHub for the Student Developer Pack), we can support these orgs on a case-by-case basis to provide them with more concurrency.

The easiest to get that done is for you to email support@travis-ci.com, referencing this discussion, and talk to our support team about the requirements and options, and to get something set up for you.

Sorry that it’s not very convenient at the moment.

Cheers,
Mathias


(Andreas Mueller) #11

Thanks for following up. I understand that this requires a larger restructuring and not just the flipping of a switch.
I have to check how thing will go with the next assignment, but I suggested to my students that they don’t use the classroom repository but their own fork.

After talking to travis CI and Github I now have 5 concurrent instances - which I know is you being accommodating.
It’s still quite a difference to the 100 that my students have on their own.

Some feedback for the github classroom folks:
I’m actually regretting to use github classroom now and I might not do so for the rest of the class.
I’m not sure it provides any benefit - it’s an interface for forking a repository, and I haven’t found any functionality beyond that. Have I missed anything?
However, it comes with many downsides such as hard time integrating with any services, and it doesn’t even implement something as simple as ENDING AN ASSIGNMENT or collecting submissions.
It would be great if you could provide some insight into what you think is the value added by github classroom. Maybe I don’t understand it. It seems to be the same as sending out a link to the forking URL, only that then the repository would be under the students account and all these issues wouldn’t arise.


(Vanessa) #12

Hi @amueller, thanks for being an early user of Classroom.

At the moment it’s a lightweight tool that handles repository creation and distribution of starter assignments.

Testing tools & assignment collection are features we hope to offer in future iterations.

I see you’ve opened a few issues to add features, which we appreciate. Ideas welcome.

Thanks for your patience as we continue work on this open source tool.


(Joey Freund) #13

I teach a course, at the University of Toronto, where we use GitHub and Travis CI.
I use Gitomator’s command-line tools to enable/disable CI (Gitomator is a set of command-line tools that I built, with the goal of being a “Swiss army knife for GitHub organizations”).


(Joanna Klukowska) #14

How was the performance with the 5 concurrent instances?

I am getting ready for a Data Structures class in Java with over 130 students in the fall and I am wondering if setting up the class with Travis is the right move.

Any feedback from those who used Github repositories with Travis ci in large classrooms would be greatly appreciated.


(Andreas Mueller) #15

It was a bit of a problem for some of the exercises. It depends really on what you want to do. I wanted to make the students configure travis themselves. I don’t think the was github classroom is set up is conducive to that. I’m not sure if I’ll use it again next year, or just have students use their own repositories instead.

Students were using their repositories but then sometimes not syncing with the classroom one, which created issues for grading.

If travis is only for grading and not meant for the students to use, it might be less of an issue.

A bigger issue was maybe that there is no easy way to enforce a deadline. There is a button to take students permissions away entirely, but that means they can’t even see their submission, which also created issues with grading.


(Lindsay Ward) #16

(A bit late reply).

I’m looking forward to more functionality with Classroom, but for now, for me, the main benefit is having students’ work in private repositories that I and my markers can access.
It’s much easier both for individual and for group assignments than getting students to create private repos and adding the correct collaborators.
Perhaps if you don’t want your assignments to be private, this is not as important.

(I’m still keen to learn more about CI and automated marking/testing…)


(Dan Wallach) #17

I’m doing the GitHub/Travis-CI integration in a class with 165 students. With only one concurrent build instance, and roughly two minutes of total compute for Travis to construct everything and run the unit tests, you can see how the latency can go through the roof near deadlines.

I’ve actually come around to the idea that this isn’t a bug. It’s a feature. The unit tests and everything else (CheckStyle, ErrorProne, etc.) running on Travis-CI are exactly the same as the tests running on their local machine. But the latency on your local machine is small, whereas the latency on Travis can be much longer. As such, I tell the students that they’d better run their tests locally, and the Travis thing is for “peace of mind” to make sure they didn’t forget anything. It’s also, of course, incredibly handy for our graders.

Of note, I’ve had a few students use the “edit” buttons on GitHub and get into a confusing-for-them world of git conflicts that our TA staff have to resolve. If Travis were to respond quickly, some student might be tempted to actually use GitHub’s edit button to do all their work. With Travis responding slowly, the students are incentivized to work out whatever problem might have driven them to want to edit directly on GitHub in the first place.

We’re just using GitHub as a mechanism for checking out and committing solutions. No cross-student collaboration. No pull requests. Etc. In a class that really exercises all the Git / GitHub collaboration features, the Travis latency would be a much bigger deal and I’d be more annoyed at the one-concurrent-build restriction.


(Andreas Mueller) #18

How about having the students create private repositories and make them do pull requests against a shared private repository? Or would that allow them to see each others work?

How do you enforce homework deadlines? Having the TAs pull all the repositories automatically at the deadline?


(Lindsay Ward) #19

I’m not using PRs as part of assessment at this stage. They do it as a code review exercise, but it’s not an assignment requirement.

My students submit a zip file of their projects to our learning management system (LMS), so I handle deadlines by marking what they submit via that - but look at GitHub to assess their use of version control.
I mark their work in my IDE with comments in my LMS, so GitHub is a small part and they could do the whole assignment without using it actually (but not get full marks).


(Dan Wallach) #20

If you’re dealing with beginning students, you have to worry about some fraction of them getting the permissions wrong and leaving their work in a public repo for all the world to see. If you use the GitHub Classroom setup, you can ensure that all of the student repos, by default are private.

GitHub Classroom has a brand-new feature where you can set a deadline and they produce links to the latest commit that was present on GitHub’s servers at the time of the deadline, thus sidestepping the problem where commit times could be forged and push times aren’t normally available without digging into the GitHub Events API. This is much easier.