SFML community forums
General => SFML development => Topic started by: eXpl0it3r on January 26, 2019, 06:56:57 pm
-
Some years ago the SFML Team had decided to remove the binaries of SFML's dependencies from the git repository. Reasons for the removal range from bad practice to bloating the size of the repository, but there were also concerns raised about usability of SFML on Windows.
The decision back then was to move the binaries into a dedicated repository and add that as sub-module. Unfortunately as you all know, that decision was never implemented.
Now, when we look ahead a bit and see how SFML may evolve in the near future, there are chances that more dependencies will be introduced (harfbuzz, Opus, MP3, ...), more platforms will be supported in one way or the other and it becomes pretty clear, that the current setup doesn't really scale well.
Having to provide binaries for all the dependencies and all the platforms that need it and keeping them updated, is a time consuming task. We also end up making trade-offs for small file size, to not further bloat the repository size. As such, the idea came up to propose yet another approach.
Remove the binaries from the git repository, but instead include the source code of said dependencies and instrument our CMake build system to build the dependencies when building SFML itself.
This has the advantage that we cut down the size of the repo, remove the binary building maintenance cost on our side, can add more dependencies without mentioned concerns and if people want to, they can more easily modify or update the dependencies for their own needs.
Notice, the goal isn't necessarily to add sub-modules or let CMake dynamically fetch stuff, but instead provide everything that's needed to build SFML and its dependencies, but we'd like to hear your ideas on that topic as well. As an example take a look at Qt's setup (https://github.com/qt/qtbase/tree/5.12/src/3rdparty).
What are your opinions on this topic and are there any takers for implementing the CMake scripts changes? :)
-
I'll help as much as I can.
Personally I think it would be a good idea to either use git submodules or cmake to maintain the dependency source code (I'd prefer the latter). They would both allow easier dependency updates, make it easier to identify which version of the dependency is being built, and make it easier for contributors to build against different dependency versions if needed
The only down side would be if the user clones the repo (and forgets to clone with submodules) then goes offline they will not be able to build SFML, but I think this is a rather specific and rare case
-
Same, will be happy to help. I think submodules + CMake's ExternalProject would be really nice to use, no other package manager needed.
Nice article to read and think about: https://blog.kitware.com/cmake-superbuilds-git-submodules/
-
So we have two voices for submodules. Any other comments on the topic, maybe especially other SFML Team members? :D
-
Updating submodules can be tedious, although here it seems we have a simple setup and won't need to update them often. But keeping this in mind, wouldn't it be possible (and not harder) to always build (missings) dependencies and throw the submodule idea out of the window? Going even further, for macOS, I would be happy if we do nothing and let the user provide the binaries (brew/macports provide all we need, at no cost for us).
-
Relying on the user to provide the binaries assumes the correct version will be available. I'd prefer for this to be an option rather than the default (Applies to all platforms, as vcpkg can easily provide all dependencies on windows too). We also already have a "USE_SYSTEM_DEPS" flag for exactly that reason
Also updating submodules shouldn't be tedious at all - it's just a small handful of git commands and shouldn't take more than a few minutes
I think submodules + CMake's ExternalProject would be really nice to use
ExternalProject can be given a git repo and commit to checkout, so there would be no reason to use the submodules. I see these options:
- Use git submodules and just add_subdirectory() in cmake.
- Use ExternalProject which does a git checkout and build when sfml is build
- Use FetchContent and add_subdirectory
Personally I like option 3 best, but I think FetchContent is only available in cmake 3.11 so that would require a version bump.
Options 1 and 3 allow the dependencies to be configured at the same time as SFML is configured, whereas option 2 would configure the dependency at build time so you'd need to hardcode some values in the cmake configuration
-
FetchContent is very nice, but having CMake 3.11 as minimal version is a bit too strict, I think. For example, Ubuntu 18.04 has CMake 3.10 only... So in our case I'd suggest using submodules + add_subdirectory
However, in some bright future it'd be nice to have FetchContent for everything.
-
Do we know what the effect on build time would be?
Since a lot of the dependencies are C libraries, their binaries have a unified ABI. With the current approach, the advantage is that they do not need to be re-compiled for each compiler and configuration (like compiler vendor, version, static/dynamic link, Release/Debug, static/dynamic runtime). So a CMake-based setup should be smart to not needlessly compile dependencies again and again.
Being C libraries usually also means the binary needs to be maintained once per platform, not per compiler.
-
It's hard to predict, but if it's really mostly C libraries, it should be negligible. CMake setup is very easy to make smart and only take Debug/Release/RelWithDebugInfo into account (they're different for C libraries too, because this usually changes optimization levels and if debug info is preserved or not).
I'd suggest by starting with some good dependencies, which have CMake-based build and experimenting from here. Can someone give me an idea where to start from?
-
You could start with FreeType, which is independent of all the vorbis jazz and AFAIK has CMake support. :)
-
I'd prefer just dumping all the C deps sources into the main repo. I can think of no upsides but only downsides to using submodules.
With submodules a simple git clone no longer clones the repo 'properly', a single commit/checkout is no longer all that you need to build SFML as it was in a given point in time and so on. It's not obvious at first what's even going on, what commands to use to get all the files in place, etc.
These C sources will be treated exactly like the binaries are now - dumped in there, used in configs where they are needed and updated once in a while. The advantage of few repos all pointing at one lib repo they all use doesn't apply here either since SFML is the sole 'end code' repo using all these libs.
Remove the binaries from the git repository, but instead include the source code of said dependencies and instrument our CMake build system to build the dependencies when building SFML itself.
This is exactly what I'd advise ('include the source code' as in - into the single SFML repo, no submodules).
This has the advantage that we cut down the size of the repo, remove the binary building maintenance cost on our side, can add more dependencies without mentioned concerns and if people want to, they can more easily modify or update the dependencies for their own needs.
Exactly. In (unlikely) case a change to a dep lib is needed there will be no wondering how main repo, submodules, commits, forks and so on all interact nor chasing submodule repos commit logs to find out who, when, where and why changed a thing (and what SFML commit said to reference that new commit's hash via the submodule). It'll all be a single SFML commit history instead.
Also, I assume since the binaries are gonna remain in the git history it'll be more like stopping the growth of repo size from each time new binaries appeared, not making it smaller, but it's still good.
-
I'd prefer just dumping all the C deps sources into the main repo. I can think of no upsides but only downsides to using submodules.
With submodules a simple git clone no longer clones the repo 'properly', a single commit/checkout is no longer all that you need to build SFML as it was in a given point in time and so on. It's not obvious at first what's even going on, what commands to use to get all the files in place, etc.
I understand that, however, we can easily add a check in CMake which will say if submodule init was performed or not. And to do submodule init is an extra command. I don't think that you really need to build SFML from fresh state very often...
Storing source as is has it's problems - mainly git history being lost + not being able to know at which exact state was the repo cloned. It's harder to update things - it will really pollute history with long diffs. And updating submodule version can be done much cleaner.
I think that storing sources as is only works for small libraries, but FreeType, for example, is pretty huge.
I will research this topic and see how other libraries with lots of dependencies handle this - maybe I'll find their experiences and arguments about using/not using submodules.
Also, I assume since the binaries are gonna remain in the git history it'll be more like stopping the growth of repo size from each time new binaries appeared, not making it smaller, but it's still good.
If you do a clone with --depth=1, the size will be much smaller without the binaries. The history won't be downloaded.
-
STB updates (2 years ago) produced huge diffs. No one ever mentioned that being a problem in any way and they are neatly stashed in a 'STB updated to x.yy' commit by themselves. Ideally no commits other than 'updated some lib or other to some version' will ever touch a given lib's dir. If a lib needs to be patched urgently because it derails SFML I'd rather it be done inline in the main repo too.
Most people using git might not even know submodules exist and if they do they will often not know the command off the top of their head (I fit into latter category). It's 1 extra command and more CMake checks for no real benefit to anyone.
The size of FreeType doesn't matter. Bigger (and less compressible) binary files are in the repo already. Amount of files doesn't matter either, git handles it, it's not some prohibitive amount (like thousands of files) and they will be stashed away in own dir and only touched by some 'FT updated to X.YY' commits. To build SFML you'd need to fetch it all anyway most of the time.
What do we care for git histories of deps? These libraries are ostensibly stable API solid C libs that will be updated rarely to some given x.y.z stable release version (so just how binaries were now except not in 5 copies of each anymore and without needing to build it all), not kept up to master or their repo where we need to debug them ourselves.
Even Qt repo linked in the first post doesn't use submodules for the third party libs. They dump all third party code into one repo and add a file saying what version it is in the root of their dirs. They discard all the history and produce huge diffs. I've seen them use submodules in some other repo (with chromium submodule for web engine and with their 'super repo' that is all submodules, maybe more) but that makes more sense for them than for SFML.
It's one more parameter or command (that most people don't know off the top of their head and even ones who know it might forget it exactly since it's such a rare operation to clone and build SFML) and extra checks in CMake, all for no real benefit/solving no real problem. These source dirs should be treated exactly like the binaries are now (except for there being like 5 version of the binaries) - totally opaquely, just changed once in a while in their own commit and used in builds (to build lib that then gets linked/installed) if the user doesn't check that 'use system deps' flag.
The reason why binaries are gonna be removed is not to ever work on these libs code but to make it more convenient to build SFML everywhere so adding submodules throws that small gain out the window instantly. Binaries were never a "I don't know what this binary's git history is" problem so just dumping C into the main repo is no worse from that standpoint either.
-
Okay, fair enough, these are some strong arguments! I'll try to include FreeType's code into SFML's repo and see how to build it conveniently.
One more thing - do we want to have an option to build dependencies statically vs dynamically? Only OpenAL requires us to build dynamically, but maybe we can give users ability to choose if they want to build SFML linked to system's dynamic binaries?
-
If user wants to link to system's dynamic binaries then doesn't it mean the bundled deps aren't built at all/CMake doesn't generate their projects/makefiles?
Is that related to SFML_USE_SYSTEM_DEPS? What does that one even do? If I check it in CMake with VS 2017 the Audio still references ogg, flac and others and Graphics references FT.
Right now the deps are static libs, get linked into dlls but not static libs (they were a long time ago but it was changed and now user must link them when linking SFML statically).
-
Yeah, I think it'll be cool to handle SFML_USE_SYSTEM_DEPS and not build dependencies, but otherwise I see that they indeed need to be statically linked into SFML's libs.
-
RE SFML_USE_SYSTEM_DEPS - I agree, this is also what I suggested earlier.
I still would prefer to use git submodules over just copy/pasting the code into the repo. I will concede it adds a tiny bit of complexity for the end user, but I think it makes it easier for contributors, particularly for maintenance and testing features/bugs in different versions of dependencies. Other than having to add an extra parameter to your first clone of the repo, all the downsides of submodules are also present when just including the raw source
I don't feel particularly strongly about it though, so I'd rather just decide on a direction so we can start implementing something
For reference, freetype2 takes <8 seconds to build on my laptop, I doubt there's much deviance from that
Update: Having gone back to the work I've done previously on this, one of the larger issues is handling the target exports as we need all the dependency targets to be part of the export target, assuming we use add_subdirectory. This might require a small amount of patching for each dependency
Another update: Proof of concept using submodules here if anyone wants to check it out https://github.com/JonnyPtn/SFML/tree/build_deps .Having a slight issue on macOS where there's an openAL thread chewing up all the CPU (any ideas on that?). Otherwise, it's working pretty well
-
To me either way would work, as such I've simply added a poll, so hopefully those who didn't want to say something, can still weigh in with their vote. :)
-
I'm with FRex on this. In my personal experience, Git submodules were a pain every single time I had to do more with them than just git clone --recurse-submodules.
Before we go this route, we should really be clear where we see the advantage over including the source.
Some disadvantages were already mentioned:
- Less user-friendly for every SFML user due to extra setup step (you can expect 100s of forum posts "why does my code not compile")
- We introduce a third-party dependency. If that third-party repository goes down, SFML will not be possible to build, with no backup (rolling back to previous version past now won't work)
- It's not even clear if all the dependencies we currently use have official and stable Git repos. If not, we would need to provide a mirror (i.e. extra maintenance).
- There's quite some hidden complexity in the Git Submodule design, see here (https://git-scm.com/book/en/v1/Git-Tools-Submodules#Cloning-a-Project-with-Submodules). This can lead to unnecessarily complex workflows for both maintainers and users.
There's also different mechanism to version external dependencies: git-subtree.
Its usage and advantages over git-submodule are explained nicely in this StackOverflow answer (https://stackoverflow.com/a/33579069).
I've personally used it (or its predecessor, before it was in Git) to pull Aurora into Thor, and it worked very well.
And I would appreciate if the decision were made based on technical arguments, not polls from a handful of people that happen to be around at the time the matter is discussed ;)
-
Git submodules were a pain every single time I had to do more with them than just git clone --recurse-submodules.
You wouldn't have to do anything more with them than just git clone --recurse-submodules
Less user-friendly for every SFML user due to extra setup step (you can expect 100s of forum posts "why does my code not compile")
It can be made extremely simple. We can use cmake show a warning if the user hasn't cloned the submodules, telling them exactly which command to run. We can also use cmake to actually check out the submodules if the user hasn't already
We introduce a third-party dependency. If that third-party repository goes down, SFML will not be possible to build, with no backup (rolling back to previous version past now won't work)
Given all the repositories except one are hosted on github, the likelihood of this being an issue is almost nil. If it's a concern we can also easily mirror them in repos we have control over
It's not even clear if all the dependencies we currently use have official and stable Git repos. If not, we would need to provide a mirror (i.e. extra maintenance).
They all have official git repositories, and even IF we were to mirror them (which I don't think we need to do) the maintenance would be equal to maintaining the source in the SFML repo (i.e. almost nil)
subtree looks interesting. Haven't tried it myself but worth considering. I'd be interested to see and try a working example of it
And I would appreciate if the decision were made based on technical arguments, not polls from a handful of people that happen to be around at the time the matter is discussed
I think we've pretty much already covered all the technical arguments, there's not really too many facets to discuss. At this point it comes down to personal preference, so a poll seems reasonable.
In the interest of fairness, here are the submodule advantages that were already mentioned:
- Better visibility over dependency versions
- Easier testing of newer dependency versions (just git checkout/pull the revision you want to test)
- Easier access to dependency history for bisecting/debugging/testing
- Less risk of human error when updating or changing dependencies
- Easier to contribute changes upstream
As I said before I don't think there's a clear technical reason to choose one or the the other (They are almost identical in practice) so we just need a decision made. It's super easy to change and the implementation is pretty much done. We just need a decision so this doesn't turn into another dead discussion
-
OK, thanks for listing it clearly! So I think it boils down to decide what's more important for SFML:
- git-submodule: have full traceability / version switching and the option to contribute upstream, at the risk of having the build break because of unavailable 3rd-party repos and extra inconvenience when cloning and updating SFML.*
- git-subtree: have self-contained and always-available repo that retains current easy workflow for users, at the cost of extra manual work for SFML developers in case they want to try different versions or contribute upstream.
Since I would argue that it's more important to make life easier for SFML users (the "S"), and that the cases we're discussing -- trying different versions of a dependency and contributing upstream -- occur rarely enough, git-subtree is more suitable in my opinion. I'm not a fan of imposing extra constraints and a more complex workflow for the average user, just to make developer life slightly easier in one of those rare scenarios. We haven't had those options until now and it was never a severe restriction.
Edit: the problem with polls is that the voters don't represent the users being affected. The average SFML user isn't necessarily active in this forum, and while adapting our Git workflow to use submodules might be an easy change for us active members/contributors, it isn't for them -- so we have to keep those use cases in mind. We also have to consider that using git-submodule is a breaking change for everyone that has a more or less automated workflow (bash/batch scripts that update and build SFML).
______
* git submodules also need special commands when pulling SFML, not just for the initial clone. This is another case where doing a normal git pull will leave dependencies without update, i.e. another source of error.
-
Subtree looks like a reasonable compromise, although information on it is a little thin spread.
We also have to consider that using git-submodule is a breaking change for everyone that has a more or less automated workflow (bash/batch scripts that update and build SFML).
It doesn't have to be - the submodules can be cloned/updated as part of the cmake script
Ultimately I don't mind too much, I'm just keen to see a decision made, if you have a better suggestion than a poll to get this done that's absolutely fine by me.
-
What about repo tool (https://gerrit.googlesource.com/git-repo/ (https://gerrit.googlesource.com/git-repo/))? Google use this to manage all the 10's of external repositories dependancies of the Android open source project. It seems to be working pretty okay for them.
I've never used the tool myself but I've heard it does a better job than submodules at dealing with multiple git repo's.
Here's a review of the tool (sorry for non french speakers) : http://www.yoannsculo.fr/git-repo-outil-gestion-multiples-repositories-arracher-cheveux/ (http://www.yoannsculo.fr/git-repo-outil-gestion-multiples-repositories-arracher-cheveux/)
-
Personally I don't think it's a good idea to add another requirement/dependency like that just to manage our dependencies
-
I also think we shouldn't overdo it. The original problem of this thread were binaries for dependencies, which need to be maintained by SFML, and where source code would have some advantages.
-
thank you for sharing the dependencies of C libraries, and advantages of not recompiled for each compiler.
-
Found one more interesting case against using git submodules - when you do release on GitHub, zip with source doesn't include ".git" subdirectory, so you can't checkout 3rd party lib sources. So yeah, you get "source dir" which you can't even build because submodules are not checkout out there.
-
Don't suppose there's been any decision either way on this?
-
I also vote for the exclusion of the bundled libraries (either in binary or source form). It's not the purpose of SFML to do this on behalf of the SFML user.
Also, it's just to complex to bring because of:
- different toolchain versions
- different architectures
- different compile options
- and so on
This is the responsability of the user to provide its dependencies, not SFML one. Less code, less build, less problems :-).
Qt did ship bundled libraries as well but they eventually decided to remove in Qt 6.