At first I was compiling the IMGUI library as a unity build with my own code. It was taking upward of five seconds each compile. So I decided to pull it out into its own library image. There are two main kinds of libraries: static and dynamic. Static libraries are essentially archived ‘object’ files (.o). The library file will have an extension of .a on mac and linux and .lib on windows. When you compile them into your own code, the linker outputs the actual code into your executable. This means, you can’t seperate out the library from your program.
The benefits are that you will never have any dependency issues. You can’t not find your own executable. The downsides to this are wasted computer resources. If multiple programs are using the same library, why not load one version of the library into RAM and everyone shares that one instance. This is where dynamic libraries come in.
They were designed to reduce the resource load on the computer. Dynamic libraries are found in /usr/lib or usr/local/lib on mac and linux, and C:\Windows\System32 on windows. They have the extension .so on Linux, .dylib on mac and .dll on windows. These are the files you link to in your build script. When you do link to them, the linker will only place the name of the library not the actual code into your executable. Then when you go to run your executable, the runtime linker will look up where the library is located and load it into RAM. And if another program is already using it, just reference that instance.
This method sounds appealing, your not duplicating preexisting code into RAM. However there are a few caveats with this method. Since the library is no longer part of the executable, we have to make sure the library exists on the user’s machine or we ship our software with the library. This can run into issues. One example is versioning. Say program A uses lib1.0 and then program B uses lib1.1 version. When we install program B, we will update lib1.0 to lib1.1. However if the library isn’t backward compatible, we have now broken program A. This is just one case of what is referred to as DLL hell. However with strict versioning, most problems are avoided.
The main case is just make sure the library you linked to exists on the end user’s machine. I found out this is not as simple as it sounds. As I was building IMGUI as a library, I wasn’t sure how to reference it. If i just said the output file was called ../bin/libimgui.dylib, my program couldn’t find it if I double clicked on my executable (i.e. not on the command line). However if I gave it an absolute path it could find it: /User/Oliver/myapp/bin/libimgui.dylib . I thought I had solved the relative look up, but soon realised people are not all called Oliver, and will store the app in all different places.
My inquiry further deepened as I was trying to link with SDL. As a test I moved the sdl.dylib out of the usr/local/lib folder and into my build folder of my program Theoretically I thought I could just reference the version in my build file. It linked successfully at _compile time_ but when I went to run the program it said image not found, and was still looking for the version in my /usr/local/lib folder.
After a lot of confusion, testing and research I learned a bit more about compiling and linking to libraries. There are two points at which the executable will look for the library: compile time and runtime. At compile time you can reference the library at any location, either with an absolute path name or with the -L /directory -lname flags. _However_ the name of the library put into your executable is the install name of library, which is a property of that library. You can’t arbitrarily change it when linking. The install name was set when the library itself was compiled. Then at _runtime_ the linker uses the _install name_ to look for that library.
The catch is that most libraries install names are absolute, meaning they always have to be in the same place on all computers. That place most likely being /usr/lib on mac and linux. So even if you ship with sdl.dylib with your exe, your exe will be looking in the wrong place.
You can see the install name of libraries you are linking to by running
1 | otool -L my_exe_name |
1 | -lld my_exe_name |
After some research I found that libraries can be compiled with a relative name. You just have to set it at the _libraries_ compile time. So for my IMGUI library I used the build script (NOTE: this was on mac, so will be a bit different on linux and a lot different on windows):
1 | clang++ -shared -install_name @rpath/../libimgui.dylib imgui*.cpp -o libimgui.dylib |
Then when I built my program I used the script:
1 | clang++ -Wl,-rpath,@executable_path/ main.cpp —limgui o my_program |
To get the relative library reference was a two step process. I first made my library have an install name prefixed by @rpath. This is a special token which will get replaced a runtime linking by a list of paths the executable has.
Secondly I add the @exectuable_path token to the programs rpath. This allows the program to know it can look in the executable’s path whenever looking for a library prefixed by @rpath.
What I found weird was that most libraries are compiled with an absolute path, which means you can’t just move the dylib your were using on your computer and into a relative folder when you distribute your product. One way around this is using the install_name_tool to manually override the install name in _your_ executable.
An example of this (on mac) is:
1 | install_name_tool -change /usr/local/lib/libSDL2-2.0.0.dylib "@executable_path/../libSDL2-2.0.0.dylib" ./myexe |
However on mac, frameworks (a bundled dynamic lib), do have relative install names, so you _can_ just move the framework into your local exe folder, as long as you have added @executable_path to the exe’s rpath at compile time.
I found after wrestling with libraries, all I was trying to achieve was what a static library provides. You know your program will have the required code to run on an end user’s machine. I think that’s why the header file libraries (eg. Sean Barrett’s stb libraries) are so popular, they are essentially a form of a static library: they’re never going to not be found.
Jon Blow mentioned this in is talk on library design for his new language. From what I understand, in the language, libraries will be compiled into the exe, so there will never be a dependency problem. I like the simplicity and ease of this approach. I think for most 3rd party libraries this should be the standard for C programming as well.
Note: I am using a mac and the scripts I put up are for mac. I don’t think Linux has the @executable_path token, instead you have to use $ORIGIN, but I haven’t compiled on linux. I haven’t investigated Windows yet. Maybe the MinGW compiler is similar.
Any thoughts, comments, fixes or more info please leave below. I hope this helps anyone else trying to link with libraries.
Thanks for reading,
Oliver
