Welcome, Guest. Please login or register. Did you miss your activation email?

Author Topic: Removing dependency binaries  (Read 289 times)

0 Members and 1 Guest are viewing this topic.

eXpl0it3r

  • Moderator
  • Hero Member
  • *****
  • Posts: 9084
    • View Profile
    • development blog
    • Email
Removing dependency binaries
« on: January 26, 2019, 06:56:57 pm »
Some years ago the SFML Team had decided to remove the binaries of SFML's dependencies from the git repository. Reasons for the removal range from bad practice to bloating the size of the repository, but there were also concerns raised about usability of SFML on Windows.
The decision back then was to move the binaries into a dedicated repository and add that as sub-module. Unfortunately as you all know, that decision was never implemented.

Now, when we look ahead a bit and see how SFML may evolve in the near future, there are chances that more dependencies will be introduced (harfbuzz, Opus, MP3, ...), more platforms will be supported in one way or the other and it becomes pretty clear, that the current setup doesn't really scale well.

Having to provide binaries for all the dependencies and all the platforms that need it and keeping them updated, is a time consuming task. We also end up making trade-offs for small file size, to not further bloat the repository size. As such, the idea came up to propose yet another approach.

Remove the binaries from the git repository, but instead include the source code of said dependencies and instrument our CMake build system to build the dependencies when building SFML itself.
This has the advantage that we cut down the size of the repo, remove the binary building maintenance cost on our side, can add more dependencies without mentioned concerns and if people want to, they can more easily modify or update the dependencies for their own needs.

Notice, the goal isn't necessarily to add sub-modules or let CMake dynamically fetch stuff, but instead provide everything that's needed to build SFML and its dependencies, but we'd like to hear your ideas on that topic as well. As an example take a look at Qt's setup.

What are your opinions on this topic and are there any takers for implementing the CMake scripts changes? :)
« Last Edit: January 27, 2019, 09:54:51 am by Laurent »
Official FAQ: https://www.sfml-dev.org/faq.php
Nightly Builds: https://www.nightlybuilds.ch/
——————————————————————
Dev Blog: https://dev.my-gate.net/
Thor: http://www.bromeon.ch/libraries/thor/

Jonny

  • Jr. Member
  • **
  • Posts: 98
    • View Profile
    • Email
Re: Removing dependency binaries
« Reply #1 on: January 26, 2019, 08:22:03 pm »
I'll help as much as I can.

Personally I think it would be a good idea to either use git submodules or cmake to maintain the dependency source code (I'd prefer the latter). They would both allow easier dependency updates, make it easier to identify which version of the dependency is being built, and make it easier for contributors to build against different dependency versions if needed

The only down side would be if the user clones the repo (and forgets to clone with submodules) then goes offline they will not be able to build SFML, but I think this is a rather specific and rare case

Elias Daler

  • Hero Member
  • *****
  • Posts: 539
    • View Profile
    • Blog / Re:creation
    • Email
Re: Removing dependency binaries
« Reply #2 on: January 27, 2019, 09:07:14 am »
Same, will be happy to help. I think submodules + CMake's ExternalProject would be really nice to use, no other package manager needed.

Nice article to read and think about: https://blog.kitware.com/cmake-superbuilds-git-submodules/

eXpl0it3r

  • Moderator
  • Hero Member
  • *****
  • Posts: 9084
    • View Profile
    • development blog
    • Email
Re: Removing dependency binaries
« Reply #3 on: January 30, 2019, 10:10:46 am »
So we have two voices for submodules. Any other comments on the topic, maybe especially other SFML Team members? :D
Official FAQ: https://www.sfml-dev.org/faq.php
Nightly Builds: https://www.nightlybuilds.ch/
——————————————————————
Dev Blog: https://dev.my-gate.net/
Thor: http://www.bromeon.ch/libraries/thor/

Hiura

  • Moderator
  • Hero Member
  • *****
  • Posts: 4325
    • View Profile
    • Email
Re: Removing dependency binaries
« Reply #4 on: January 30, 2019, 10:24:53 am »
Updating submodules can be tedious, although here it seems we have a simple setup and won't need to update them often. But keeping this in mind, wouldn't it be possible (and not harder) to always build (missings) dependencies and throw the submodule idea out of the window? Going even further, for macOS, I would be happy if we do nothing and let the user provide the binaries (brew/macports provide all we need, at no cost for us).
SFML / OS X developer

Jonny

  • Jr. Member
  • **
  • Posts: 98
    • View Profile
    • Email
Re: Removing dependency binaries
« Reply #5 on: January 30, 2019, 02:00:38 pm »
Relying on the user to provide the binaries assumes the correct version will be available. I'd prefer for this to be an option rather than the default (Applies to all platforms, as vcpkg can easily provide all dependencies on windows too). We also already have a "USE_SYSTEM_DEPS" flag for exactly that reason

Also updating submodules shouldn't be tedious at all - it's just a small handful of git commands and shouldn't take more than a few minutes

Quote
I think submodules + CMake's ExternalProject would be really nice to use

ExternalProject can be given a git repo and commit to checkout, so there would be no reason to use the submodules. I see these options:

- Use git submodules and just add_subdirectory() in cmake.
- Use ExternalProject which does a git checkout and build when sfml is build
- Use FetchContent and add_subdirectory

Personally I like option 3 best, but I think FetchContent is only available in cmake 3.11 so that would require a version bump.

Options 1 and 3 allow the dependencies to be configured at the same time as SFML is configured, whereas option 2 would configure the dependency at build time so you'd need to hardcode some values in the cmake configuration
« Last Edit: January 30, 2019, 02:02:57 pm by Jonny »

Elias Daler

  • Hero Member
  • *****
  • Posts: 539
    • View Profile
    • Blog / Re:creation
    • Email
Re: Removing dependency binaries
« Reply #6 on: February 10, 2019, 12:08:10 am »
FetchContent is very nice, but having CMake 3.11 as minimal version is a bit too strict, I think. For example, Ubuntu 18.04 has CMake 3.10 only... So in our case I'd suggest using submodules + add_subdirectory

However, in some bright future it'd be nice to have FetchContent for everything.

Nexus

  • Moderator
  • Hero Member
  • *****
  • Posts: 6143
  • Thor Developer
    • View Profile
    • Bromeon
Re: Removing dependency binaries
« Reply #7 on: February 11, 2019, 08:15:39 am »
Do we know what the effect on build time would be?

Since a lot of the dependencies are C libraries, their binaries have a unified ABI. With the current approach, the advantage is that they do not need to be re-compiled for each compiler and configuration (like compiler vendor, version, static/dynamic link, Release/Debug, static/dynamic runtime). So a CMake-based setup should be smart to not needlessly compile dependencies again and again.

Being C libraries usually also means the binary needs to be maintained once per platform, not per compiler.
Zloxx II: action platformer
Thor Library: particle systems, animations, dot products, ...
SFML Game Development: first SFML book

 

anything