Continuous Integration

[eXpl0it3r]

The two CI pipeline help us tackle the following four challenges:

• Ensure that the code base builds successfully across all supported OS (Windows, Linux, macOS, Android, iOS)
• During development we often want people to test our branches, but it can be quite cumbersome for some to keep building SFML over and over again. By storing the artifacts of the builds we can provide up-to-date builds for each branch, including the master branch.
• Run our unit tests and in the future hopefully also our integration tests
• Detect problematic code with static code analyzers (Clang & Cppcheck)

Buildbot
Our custom CI pipeline is running with Buildbot and will automatically start build jobs whenever a branch on the SFML git repository is updated. For pull requests someone from the SFML Team will have to comment a trigger word in order to ensure that not random code is executed on the workers.

Buildbot: https://ci.sfml-dev.org/

There are four workers each having a different OS and doing multiple builds:

• Debian & Android & Static Analyzer
• Windows 10
• FreeBSD
• macOS & iOS

Additionally, we store all the build artifacts on the server. The list can be browsed either by branch or commit hash.

Artifacts: https://artifacts.sfml-dev.org/

Travis CI
For open source projects Travis CI is free, as such someone contributed a build file that also builds for all supported platforms and runs the existing unit test suite.
The big advantage here is that these builds run for every pull request, as they are executed on a containerized and secured system.
The artifacts from Travis CI are not stored as of right now

Travis CI: https://travis-ci.org/github/SFML/SFML

[Jesper Juhl]

Great. Nice job.

[Foaly]

This is really cool. I think it will definitly be helpful for testing. If you want to have a quick way of testing new stuff without having to build it yourself (maybe on a platform where you don't have a build setup...). Also its a nice way to ensure the code compiles on all platforms.

One thing that would be a nice addition is to also extend the builds to pull requests. That way developers that are not in the core team can befit from this as well.

Anyway this is really nice!

[binary1248]

One thing that would be a nice addition is to also extend the builds to pull requests. That way developers that are not in the core team can befit from this as well.

Although this is technically possible, it poses a security risk as anybody could create pull requests containing/producing malicious content due to the way it is currently automated.

[Foaly]

I see. But how big is the risk? From my understanding the code is only compiled on the machines and not run (exept for maybe unit test in the future). Also the code for generating the build is always open to see.

I'm just thinking it would be nice for developers outside the SFML core team to also benefit from that. I for example have access to a Mac at work which I could use to test. But I don't have a build setup there.

If it's possible the build for pull requests could also be triggered manually once someone has done a code review.

[eXpl0it3r]

I see. But how big is the risk? From my understanding the code is only compiled on the machines and not run (exept for maybe unit test in the future). Also the code for generating the build is always open to see.

CMake is executed, so anything is possible. You can go and call anything from within a CMake  scripts. And even if its open, the building starts as soon as a commit is made, so there's no time for anyone to intervene.
I'm not sure if it's doable to add a option for us to green-light some PRs.

[Foaly]

Ah I didn't think of Cmake...
Well it was just an idea. Maybe a manual option could be investigated. Even if not, this is still very cool!

[Klaim]

I'm interested and curious about knowing what made you choose  Buildbot. I want to setup CI too but have been pondering between Jenkins (apparently easy to setup) and Buildbot (apparently more flexible?).
Was it chosen just because the team was familiar with it or is there some other interesting reasons?

[binary1248]

We used to use Jenkins for a while... was fast to set up, but if you want to get something done that is atypical, it becomes really painful. The best part was that we couldn't (or at least didn't know how to) restart builds that failed due to various reasons. Jenkins has a plugin system, but like Jenkins itself, if you have something you really want done that isn't available as a plugin, you are almost out of luck (you will have to write it yourself if you are able).

Those are just a few of the things I can remember from the "dark days" when we used Jenkins. The others who were in charge of it (eXpl0it3r, Tank) might remember more.

Simply put, comparing Jenkins and Buildbot is kind of like comparing Windows and Linux. If you want freedom/flexibility and have the time to invest in setting up something that is perfect, then go for Buildbot. If you need something up and running really fast and are willing to cut a few corners here and there and throw future-proof-ness out the window, you can go with Jenkins. Buildbot configuration is done almost exclusively using Python. SFML's Buildbot configuration is still sitting in my repository. I haven't updated it to the currently active configuration yet (the current license also sucks ), but when I do I will also transfer the repository over to the organization.

[eXpl0it3r]

One other issue that just came to my mind is that we couldn't nicely package the artifacts like we do now.
Or simple things, like "rebuild master for every new commit, regardless if the commit was already built once".

Python just gives you the full power to do whatever you want however you want it. While Jenkins can only do what you want if you write plugins for it.

[Klaim]

Thanks for the input, it clarify a lot the "flexibility vs simplicity" feedback I got so far.

[eXpl0it3r]

This thread hasn't seen some update in quite a while, despite the infrastructure changing quite a bit.

I've updated the first post to include the Travis CI pipeline and fixed some links.

Additionally, note that we have now VS 2019 builds for both Travis CI and Buildbot since last weekend.
You can find the artifacts for is with the label windows-vc16: https://artifacts.sfml-dev.org/by-branch/master/