Linking dependencies

[Tank]

Hi guys, hi Laurent,

I wanted to ask (again) about changing the way SFML links to dependencies. It has been discussed a bit in this thread already: http://en.sfml-dev.org/forums/index.php?topic=7602.0

When talking with some guys in #SFML (irc.boxbox.org) and ##MinGW (Freenode), and remembering the issues we had with SFGUI (double definitions at the linker stage, i.e. importing GLEW 2 times), I felt like bringing this back to the table again.

Let's identify the problem first: SFML itself links to libraries it's using, also internally only. Those libraries are provided as pre-built static libraries in the releases and also in the source snapshots/SCM.  When building SFML, it links those dependencies in, statically, i.e. the archives (/static libs) are packed into the final SFML library.

I think the main purpose of that is to ease using SFML, so that the user doesn't has to link to GLEW, OpenAL and all the other dependencies.

However, there are some problems that shouldn't be ignored and that can lead to severe errors and crashes under certain circumstances: If the target executable (i.e. the user program) imports the same dependecies that are already being defined in SFML, clashes might happen (especially when using different GLEW versions). Also the developer can't exchange the dependency, e.g. to upgrade to a later version. Both together can lead to severe problems when, for example, the ABI changes. In general the ODR (one-defition rule) seems to be violated.

The solution would be to throw away all target_link_libraries() calls in SFML's CMake scripts. However I'm not 100% sure if that works with MSVC too. Such a change would indeed require the end-user to link to all the deps himself, but in my opinion telling/teaching them the right way is better than going for an easy approach with accepting that fatal bugs might happen.

[Laurent]

The solution would be to throw away all target_link_libraries() calls in SFML's CMake scripts

You meant the sfml_static_add_libraries calls, right?

[eXpl0it3r]

You meant the sfml_static_add_libraries calls, right?

Specially that one but the others aren't actually need either.
If I've understood the discussion in the channels correctly the idea of static linking is that all the dependencies should be linked in the final state.

It was me that started the discussion on ##mingw because ar.exe couldn't find the OpenGL library. Somewhere along the way SFML/CMake/whoever assumes that the libs are in ../lib/ folder seen from the GCC directory which is wrong for MinGW w64 (it's located in /mingw-VERSION/lib/). This then lead to the discussion, that SFML shouldn't even touch the opengl libs.

Anyways I've removed all the target_link_libraries() and build SFML succesfully, then I had to specify all the deps in my application and everything build without any problems.

[Laurent]

But target_link_libraries is when building dynamic libraries. The thing we're discussing now, i.e. adding static libraries to SFML static libraries, is done with sfml_static_add_libraries. If you remove all target_link_libraries calls then dynamic SFML libraries will produce linker errors.

[Tank]

If you remove all target_link_libraries calls then dynamic SFML libraries will produce linker errors.

That's not the case on Linux, it just builds fine. Also with MinGW on Windows. Maybe MSVC works different here, can't comment on that.

But yes, I'm talking about packing other library's object code into SFML.

[Laurent]

That's not the case on Linux, it just builds fine. Also with MinGW on Windows. Maybe MSVC works different here, can't comment on that.

Wait... how can the linker produce a binary that uses xyz functions from a library, without linking to it? What you guys say doesn't make sense, but it's probably me missing something obvious or misunderstanding you

[Tank]

Because normally only symbols are referenced. The actual code gets linked into executable in the executable's linker step. When producing an executable, you always have to link to every single dependency, otherwise you get unresolved externals.

That's different for libraries. To satisfy ODR, libraries should not link to their own dependencies. Otherwise it might happen that both the library AND the executable link in the same object code, which can lead to serious problems like described in the first post.

Just try it: Write a source file that uses an external library's function (e.g. OpenGL) and compile and link it into a library, *without* linking to that external library. It won't produce any errors -- except MSVC works differently, I don't know if it does.

On Linux I could perfectly build SFGUI without linking to SFML at all.

[binary1248]

I think the confusing part about this whole discussion is really what is involved in this ominous "linking" process. When the linker runs different things happen depending on what kind of linkage the developer wants for his binary.

For static linkage obviously a code copy has to be done so that the final executable is portable across systems with the same architecture. This means that the calls to the static library are just resolved by the linker to internal addresses within the same executable where the copy of the code resides. This however is only the case when compiling the final executable, not for libraries, except for the special case of SFML. If you have multiple copies of the same library in 2nd order libraries ODR will horribly fail because you confused the linker and it won't know which code source to use. This is why you only provide the code of the library when the final executable is built. There the "last developer" can specify the source of the code for a specific library and the linker will know where to copy it from to satisfy everything that is needed by the 2nd order libraries and the executable. If multiple libraries all included their dependencies in themselves it would be akin to someone building multiple copies of SFML (all with different names of course) and linking them all to his final executable. Because the names of the exported symbols are all the same across those libraries something is bound to go wrong.

When linking dynamically things get tricky. In this case the linker just produces a table of external symbols that are needed (linked in) when the code is executed. This seems to be the simpler linkage type, but if you consider that you can dynamically link to a library that statically linked a library that you dynamically link in your final executable.... it starts to get really messy. You have to be really careful here in order to avoid name clashes. Some libraries probably facilitate such complex linking by inserting tons of preprocessor macros for each scenario, but seriously... if it can be avoided, why do so in the first place. I already hate mixing linkage types and it seems that doing so isn't recommended in MSVC since they have a very exotic way of doing things in general. Windows also happens to be the only OS I know of that requires (as in not optional) the developer to explicitly mark functions that should be exported out of a dynamic library. Is this a result of their vendor lock-in non-cooperative paranoid software development paradigm? Possibly, but that is a discussion for another time.

GLEW is the reason why this discussion started in the first place, and it is a result of how GLEW is built. From what I can tell, GLEW defines many many global variables that are hidden behind a few layers of macros. These global variables tell you whether capability xyz is available on the hardware as well as provide addresses to the advanced OpenGL functions we all love to use. In order to fill these variables with meaningful data they have to be initialized at some point, which is why glewInit() needs to be called. I think the problem should be obvious now. When we in SFGUI want to call glewInit() to initialize the variables in order to use them (it doesn't work if we don't call it because SFML calling it doesn't affect the variables we can access) we need a reference to the GLEW code. This is already included in sfml-graphics but we can't access it under certain circumstances, so we try to link in another copy of GLEW. This also doesn't work because the name already exists in the executable space and thus the headache begins. We ended up using GLee, just because it uses different names for exactly the same functionality.

I think it will make the lives of library developers like myself much easier if SFML dealt with dependencies "normally". The external libs would just need to be copied into the same folder as the SFML libs and

Code: [Select]

g++ main.o -o sfml-app -lsfml-graphics -lsfml-window -lsfml-systemwould become something like

Code: [Select]

g++ main.o -o sfml-app -lsfml-graphics -lsfml-window -lsfml-system -lfreetype -lglew -ljpeg -lopenal32 -lsndfilebut I'm sure that's something that won't even bother newcomers.

[Laurent]

Just try it: Write a source file that uses an external library's function (e.g. OpenGL) and compile and link it into a library, *without* linking to that external library. It won't produce any errors

Hum ok, I never noticed that. I don't know why one would do that though, except adding more work for the final user. But that's another discussion... unless you also suggest that I remove dependencies from shared libraries (which I've never seen).

The problem is clear and well understood by everyone here (I think), thanks for summarizing it so well.

we need a reference to the GLEW code. This is already included in sfml-graphics but we can't access it under certain circumstances

Can you clarify this point? Since SFML embeds GLEW, you just don't have to (or "can't" ) link your own version of it, but the result is the same, it gets included to the final executable.

I'm sure that's something that won't even bother newcomers.

The Linux example is straight-forward of course, most dependencies are already installed and are in standard paths, so it's just a matter of adding a few -lxxx. On Windows things are worse: you need to add include and linker search paths (to those extra static libs), and you need to add them in your linker settings. It's not a big deal, but for beginners who can't even follow the getting started tutorial, it will be definitely too much. And be sure to see a lot of posts concerning this on the forum.

I'm not saying I'm against the idea of handling static libs the way they should be, but a lot of people will see the disadvantages, and only a few will see the benefits.

[binary1248]

we need a reference to the GLEW code. This is already included in sfml-graphics but we can't access it under certain circumstances

Can you clarify this point? Since SFML embeds GLEW, you just don't have to (or "can't" ) link your own version of it, but the result is the same, it gets included to the final executable.

OK, imagine theoretically, SFGUI would remove all dependencies to SFML, which means SFML isn't needed to use SFGUI, SFGUI could use GLEW on it's own because we would link against it as already mentioned so many times. What would happen if someone wanted to use SFML and this new SFGUI together? It won't work because SFML includes GLEW in sfml-graphics and (just as an example) is linked dynamically. SFGUI is also linked dynamically but needs to link in GLEW at runtime. It won't work because there will be 2 GLEWs in the final executable, the one inside SFML and the dynamically linked one.

I'm sure that's something that won't even bother newcomers.

The Linux example is straight-forward of course, most dependencies are already installed and are in standard paths, so it's just a matter of adding a few -lxxx. On Windows things are worse: you need to add include and linker search paths (to those extra static libs), and you need to add them in your linker settings. It's not a big deal, but for beginners who can't even follow the getting started tutorial, it will be definitely too much. And be sure to see a lot of posts concerning this on the forum.

I'm not saying I'm against the idea of handling static libs the way they should be, but a lot of people will see the disadvantages, and only a few will see the benefits.

If the libs are in the same folder as the library files themselves and the headers in the SFML header folders there will be no extra overhead. Sure placing non-SFML headers in the SFML header folders is a bit unorthodox, but if it helps newcomers, why not. It is still less strange than including everything in the static library archive. The only extra work that has to be done is a copy from the extlibs folder into the library file destination and header folders. Since you would write the script for this the newcomer won't notice any file copying or extra compiler/linker search settings.

[Laurent]

It won't work because there will be 2 GLEWs in the final executable, the one inside SFML and the dynamically linked one.

I know that, but I was asking about the "we can't access it" part. So in fact it was just another way to refer to the "different versions linked together" problem? I thought you were talking about another problem.

If the libs are in the same folder as the library files themselves and the headers in the SFML header folders there will be no extra overhead.

I don't like the idea of mixing SFML and external libraries in the same folder. But yes, that could be another option, which is probably not worse than the current one I guess.

[eXpl0it3r]

I don't like the idea of mixing SFML and external libraries in the same folder. But yes, that could be another option, which is probably not worse than the current one I guess.

Or you could also just copy the extlibs folder. Sure the users would then need to specify two paths but I guess this could work out...

[binary1248]

I know that, but I was asking about the "we can't access it" part. So in fact it was just another way to refer to the "different versions linked together" problem?

Yeah it's all one and the same problem with many different names.

[Tank]

I don't like the idea of mixing SFML and external libraries in the same folder. But yes, that could be another option, which is probably not worse than the current one I guess.

Understandable, however it's already the same with the DLLs that have to be shipped with SFML in order for it to work, like sndfile and OpenAL. The user has to know that those must be copied to the final executable -- and I don't see a lot of postings on the forum asking why the program crashes with asking about a DLL.

Like binary said, the advantage of the .lib files is that those can be simply copied to SFML's lib directory, i.e. where the sfml-* libraries are saved to, and the user doesn't has to add any more linker and/or compiler search paths (considering that headers of external libraries also also added to SFML's include directory).

Remember that this is nothing that can be avoided really. It really leads to problems, and working around those is either not possible (in SFGUI's case) or very annoying to do (e.g. cloning SFML's source code, removing linking to external libraries, rebuilding).

Regarding the newbies: IF they are reading the tutorials -- and I think the most do if they have never used a similar library before -- then it'll be just adding a few more external libs to the linker options. A positive side-note: Many users asking "Can I use SFML with OpenGL or DirectX?" on IRC will probably recognize that SFML indeed uses OpenGL when adding "opengl" to the linker options.

[Laurent]

Like binary said, the advantage of the .lib files is that those can be simply copied to SFML's lib directory, i.e. where the sfml-* libraries are saved to, and the user doesn't has to add any more linker and/or compiler search paths (considering that headers of external libraries also also added to SFML's include directory).

But it automatically creates a conflict if the user has its own version of one library in another search path.