Removing dependency binaries

[eXpl0it3r]

Some years ago the SFML Team had decided to remove the binaries of SFML's dependencies from the git repository. Reasons for the removal range from bad practice to bloating the size of the repository, but there were also concerns raised about usability of SFML on Windows.
The decision back then was to move the binaries into a dedicated repository and add that as sub-module. Unfortunately as you all know, that decision was never implemented.

Now, when we look ahead a bit and see how SFML may evolve in the near future, there are chances that more dependencies will be introduced (harfbuzz, Opus, MP3, ...), more platforms will be supported in one way or the other and it becomes pretty clear, that the current setup doesn't really scale well.

Having to provide binaries for all the dependencies and all the platforms that need it and keeping them updated, is a time consuming task. We also end up making trade-offs for small file size, to not further bloat the repository size. As such, the idea came up to propose yet another approach.

Remove the binaries from the git repository, but instead include the source code of said dependencies and instrument our CMake build system to build the dependencies when building SFML itself.
This has the advantage that we cut down the size of the repo, remove the binary building maintenance cost on our side, can add more dependencies without mentioned concerns and if people want to, they can more easily modify or update the dependencies for their own needs.

Notice, the goal isn't necessarily to add sub-modules or let CMake dynamically fetch stuff, but instead provide everything that's needed to build SFML and its dependencies, but we'd like to hear your ideas on that topic as well. As an example take a look at Qt's setup.

What are your opinions on this topic and are there any takers for implementing the CMake scripts changes?

[Jonny]

I'll help as much as I can.

Personally I think it would be a good idea to either use git submodules or cmake to maintain the dependency source code (I'd prefer the latter). They would both allow easier dependency updates, make it easier to identify which version of the dependency is being built, and make it easier for contributors to build against different dependency versions if needed

The only down side would be if the user clones the repo (and forgets to clone with submodules) then goes offline they will not be able to build SFML, but I think this is a rather specific and rare case

[Elias Daler]

Same, will be happy to help. I think submodules + CMake's ExternalProject would be really nice to use, no other package manager needed.

Nice article to read and think about: https://blog.kitware.com/cmake-superbuilds-git-submodules/

[eXpl0it3r]

So we have two voices for submodules. Any other comments on the topic, maybe especially other SFML Team members?

[Hiura]

Updating submodules can be tedious, although here it seems we have a simple setup and won't need to update them often. But keeping this in mind, wouldn't it be possible (and not harder) to always build (missings) dependencies and throw the submodule idea out of the window? Going even further, for macOS, I would be happy if we do nothing and let the user provide the binaries (brew/macports provide all we need, at no cost for us).

[Jonny]

Relying on the user to provide the binaries assumes the correct version will be available. I'd prefer for this to be an option rather than the default (Applies to all platforms, as vcpkg can easily provide all dependencies on windows too). We also already have a "USE_SYSTEM_DEPS" flag for exactly that reason

Also updating submodules shouldn't be tedious at all - it's just a small handful of git commands and shouldn't take more than a few minutes

I think submodules + CMake's ExternalProject would be really nice to use

ExternalProject can be given a git repo and commit to checkout, so there would be no reason to use the submodules. I see these options:

- Use git submodules and just add_subdirectory() in cmake.
- Use ExternalProject which does a git checkout and build when sfml is build
- Use FetchContent and add_subdirectory

Personally I like option 3 best, but I think FetchContent is only available in cmake 3.11 so that would require a version bump.

Options 1 and 3 allow the dependencies to be configured at the same time as SFML is configured, whereas option 2 would configure the dependency at build time so you'd need to hardcode some values in the cmake configuration

[Elias Daler]

FetchContent is very nice, but having CMake 3.11 as minimal version is a bit too strict, I think. For example, Ubuntu 18.04 has CMake 3.10 only... So in our case I'd suggest using submodules + add_subdirectory

However, in some bright future it'd be nice to have FetchContent for everything.

[Nexus]

Do we know what the effect on build time would be?

Since a lot of the dependencies are C libraries, their binaries have a unified ABI. With the current approach, the advantage is that they do not need to be re-compiled for each compiler and configuration (like compiler vendor, version, static/dynamic link, Release/Debug, static/dynamic runtime). So a CMake-based setup should be smart to not needlessly compile dependencies again and again.

Being C libraries usually also means the binary needs to be maintained once per platform, not per compiler.

[Elias Daler]

It's hard to predict, but if it's really mostly C libraries, it should be negligible. CMake setup is very easy to make smart and only take Debug/Release/RelWithDebugInfo into account (they're different for C libraries too, because this usually changes optimization levels and if debug info is preserved or not).

I'd suggest by starting with some good dependencies, which have CMake-based build and experimenting from here. Can someone give me an idea where to start from?

[eXpl0it3r]

You could start with FreeType, which is independent of all the vorbis jazz and AFAIK has CMake support.

[FRex]

I'd prefer just dumping all the C deps sources into the main repo. I can think of no upsides but only downsides to using submodules.

With submodules a simple git clone no longer clones the repo 'properly', a single commit/checkout is no longer all that you need to build SFML as it was in a given point in time and so on. It's not obvious at first what's even going on, what commands to use to get all the files in place, etc.

These C sources will be treated exactly like the binaries are now - dumped in there, used in configs where they are needed and updated once in a while. The advantage of few repos all pointing at one lib repo they all use doesn't apply here either since SFML is the sole 'end code' repo using all these libs.

Remove the binaries from the git repository, but instead include the source code of said dependencies and instrument our CMake build system to build the dependencies when building SFML itself.

This is exactly what I'd advise ('include the source code' as in - into the single SFML repo, no submodules).

This has the advantage that we cut down the size of the repo, remove the binary building maintenance cost on our side, can add more dependencies without mentioned concerns and if people want to, they can more easily modify or update the dependencies for their own needs.

Exactly. In (unlikely) case a change to a dep lib is needed there will be no wondering how main repo, submodules, commits, forks and so on all interact nor chasing submodule repos commit logs to find out who, when, where and why changed a thing (and what SFML commit said to reference that new commit's hash via the submodule). It'll all be a single SFML commit history instead.

Also, I assume since the binaries are gonna remain in the git history it'll be more like stopping the growth of repo size from each time new binaries appeared, not making it smaller, but it's still good.

[Elias Daler]

I'd prefer just dumping all the C deps sources into the main repo. I can think of no upsides but only downsides to using submodules.

With submodules a simple git clone no longer clones the repo 'properly', a single commit/checkout is no longer all that you need to build SFML as it was in a given point in time and so on. It's not obvious at first what's even going on, what commands to use to get all the files in place, etc.

I understand that, however, we can easily add a check in CMake which will say if submodule init was performed or not. And to do submodule init is an extra command. I don't think that you really need to build SFML from fresh state very often...

Storing source as is has it's problems - mainly git history being lost + not being able to know at which exact state was the repo cloned. It's harder to update things - it will really pollute history with long diffs. And updating submodule version can be done much cleaner.

I think that storing sources as is only works for small libraries, but FreeType, for example, is pretty huge.

I will research this topic and see how other libraries with lots of dependencies handle this - maybe I'll find their experiences and arguments about using/not using submodules.

Also, I assume since the binaries are gonna remain in the git history it'll be more like stopping the growth of repo size from each time new binaries appeared, not making it smaller, but it's still good.

If you do a clone with --depth=1, the size will be much smaller without the binaries. The history won't be downloaded.

[FRex]

STB updates (2 years ago) produced huge diffs. No one ever mentioned that being a problem in any way and they are neatly stashed in a 'STB updated to x.yy' commit by themselves. Ideally no commits other than 'updated some lib or other to some version' will ever touch a given lib's dir. If a lib needs to be patched urgently because it derails SFML I'd rather it be done inline in the main repo too.

Most people using git might not even know submodules exist and if they do they will often not know the command off the top of their head (I fit into latter category). It's 1 extra command and more CMake checks for no real benefit to anyone.

The size of FreeType doesn't matter. Bigger (and less compressible) binary files are in the repo already. Amount of files doesn't matter either, git handles it, it's not some prohibitive amount (like thousands of files) and they will be stashed away in own dir and only touched by some 'FT updated to X.YY' commits. To build SFML you'd need to fetch it all anyway most of the time.

What do we care for git histories of deps? These libraries are ostensibly stable API solid C libs that will be updated rarely to some given x.y.z stable release version (so just how binaries were now except not in 5 copies of each anymore and without needing to build it all), not kept up to master or their repo where we need to debug them ourselves.

Even Qt repo linked in the first post doesn't use submodules for the third party libs. They dump all third party code into one repo and add a file saying what version it is in the root of their dirs. They discard all the history and produce huge diffs. I've seen them use submodules in some other repo (with chromium submodule for web engine and with their 'super repo' that is all submodules, maybe more) but that makes more sense for them than for SFML.

It's one more parameter or command (that most people don't know off the top of their head and even ones who know it might forget it exactly since it's such a rare operation to clone and build SFML) and extra checks in CMake, all for no real benefit/solving no real problem. These source dirs should be treated exactly like the binaries are now (except for there being like 5 version of the binaries) - totally opaquely, just changed once in a while in their own commit and used in builds (to build lib that then gets linked/installed) if the user doesn't check that 'use system deps' flag.

The reason why binaries are gonna be removed is not to ever work on these libs code but to make it more convenient to build SFML everywhere so adding submodules throws that small gain out the window instantly. Binaries were never a "I don't know what this binary's git history is" problem so just dumping C into the main repo is no worse from that standpoint either.

[Elias Daler]

Okay, fair enough, these are some strong arguments! I'll try to include FreeType's code into SFML's repo and see how to build it conveniently.

One more thing - do we want to have an option to build dependencies statically vs dynamically? Only OpenAL requires us to build dynamically, but maybe we can give users ability to choose if they want to build SFML linked to system's dynamic binaries?

[FRex]

If user wants to link to system's dynamic binaries then doesn't it mean the bundled deps aren't built at all/CMake doesn't generate their projects/makefiles?

Is that related to SFML_USE_SYSTEM_DEPS? What does that one even do? If I check it in CMake with VS 2017 the Audio still references ogg, flac and others and Graphics references FT.

Right now the deps are static libs, get linked into dlls but not static libs (they were a long time ago but it was changed and now user must link them when linking SFML statically).