Libraries - Your Own and The System's - Part 2

This post will be about system libraries - such as those you get in development packages in most Linux distributions - and the mechanisms used to allow you to use multiple versions of the same library.

System libraries

This is the sort of stuff that should be explained to new programmers on Linux, but doesn't really seem to be. *-dev/*-devel packages on Linux were kind of confusing to me at first, and this stuff can be tricky to explain, so if you get lost, the example in the next section should make things clearer.

As you should know by now, some (most) programs use shared libraries. These libraries expose symbols that are associated with a certain block of code or data. These symbols need to be visible at two distinct stages of linking.

The first stage is at link-time. The linker (e.g. ld) needs to be able to see some evidence of a symbol to know that it's there. It can then put information about where to get the code/data associated with the symbol into the executable. The second stage is at run-time. The dynamic linker (on Linux, usually ld-linux.so.2) has to use the information in the executable to find the appropriately named library that contains the symbol, so that it can load the associated code/data into memory (unless it's already been loaded into memory by another program...), ready to be executed.

Because of this, the shared library that the dynamic linker sees at run-time has to be compatible with the one the linker saw at link-time. If a function in the library is updated for a bugfix, everything'll work just fine (unless the program depended on the bug - a major problem on Windows, according to some Windows programmers I know), but if its API is changed (i.e. a function is redefined, or simply given a different name) then the dynamic linker won't be able to find the relevant symbol and the program will simply crash or - even worse - start executing a function that does something different that it expects, which when you think about it is unlikely to end well.

To overcome this problem, the name of the library searched for by the dynamic linker is different to the name searched for by the link-time linker*. When you specify -lX at link-time, the linker searches for a file named libX.so. For example, on my system, libm.so (the library that contains definitions of standard C mathematical functions in <math.h>, such as cos and sin) is located at /usr/lib/libm.so, so passing -lm to the linker makes it search for this file. However, libm.so is not a normal file, but a symlink. There are usually two "levels" of symlinking when it comes to shared libraries:

$ ls {/usr,}/lib/libm.so*
lrwxrwxrwx 1 root root 14 2010-01-13 09:47 /lib/libm.so.6 -> libm-2.10.1.so
lrwxrwxrwx 1 root root 14 2010-01-13 09:47 /usr/lib/libm.so -> /lib/libm.so.6

The real library is /lib/libm-2.10.1.so. This means you can develop by just linking with libm.so, and you'll automatically use the latest version of the library.

There is, of course, another aspect of the naming scheme to consider. When libm-2.10.1.so was compiled, somewhere on the (probably huge) command-line would have been an option like -soname=libm.so.6. This soname is the filename that the link-time linker should tell the dynamic linker to search for. This way, if you get a bugfix version of a library like libm-2.10.2.so, your symlink libm.so.6 will be updated to point to this newer library. All your programs that want to use libm.so.6 will now use the new code, and none of them need to be recompiled.

Additionally, you could update to libm.so.7, and programs compiled against libm.so.6 would still work so long as there's a libm.so.6 on your system - it would still point to the latest version that's compatible with version 6, and any program that wants to use version 7 is free to do so. And then by updating the libm.so symlink, all your compilations will be done against the newer version of the library.

An example

So, let's look at small do-it-yourself example. We'll use libexample.so from my last post.

The first thing we need to do is compile is our library, and I assume you have func1.o and func2.o handy from the last post. We'll compile version 1.0.1 of libexample, and we'll tell the dynamic linker to use version 1, which will then be a symlink to the latest version 1 of the library. So delete libexample.so if it's still in the current directory, and:

gcc func1.o func2.o -o libexample.so.1.0.1 -shared -Wl,-soname=libexample.so.1 -fPIC
$ ln -sv libexample.so.1.0.1 libexample.so.1
`libexample.so.1' -> `libexample.so.1.0.1'
$ ln -sv libexample.so.1 libexample.so
`libexample.so' -> `libexample.so.1'
$ ls libexample.so*
lrwxrwxrwx 1 john john   15 2010-03-23 21:54 libexample.so -> libexample.so.1*
lrwxrwxrwx 1 john john   19 2010-03-23 21:54 libexample.so.1 -> libexample.so.1.0.1*
-rwxr-xr-x 1 john john 6.7K 2010-03-23 21:39 libexample.so.1.0.1*

We now have our chain of shared libraries we need in order to use our versioning scheme. When we compile an application against the library, we tell it to look for libexample.so in the current directory:

$ gcc program.c -o program-shared -I. -L. -lexample

However, if we look at the dynamic dependencies of program-shared:

$ ldd program-shared
 linux-gate.so.1 =>  (0xb78d5000)
 libexample.so.1 => not found
 libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb776f000)
 /lib/ld-linux.so.2 (0xb78d6000)

We see it depends on libexample.so.1 - there's no mention of the more general libexample.so or the more specific libexample.so.1.0.1. So if we make an incompatible libexample.so.2, there's no problem - we redefine the symlink libexample.so to point to libexample.so.2 so that any new compilations take place against our updated library, but old programs compiled against libexample.so.1 will still be able to find it, and will still work. Plus, if we update libexample.1.0.1 to libexample.so.1.0.2, we can simply redefine the symlink libexample.1 to point to libexample.1.0.2, and programs using the old library will use new bugfixes.

How Linux development packages are arranged

So, now that you've got the general principle of this method, understanding Linux development packages should be a cinch.

When a user gets the standard package, they get all the versioned library files. The Debian libssh-2 package here, for example, will give you the following shared library files:

• /usr/lib/libssh.so.2


• /usr/lib/libssh.so.2.0.0

While the development package just gives you:

• /usr/lib/libssh.so

Along with all the header files you need and a static version of the library.



My next post will cover the behaviour of the dynamic linker (including how you can change it) such as where it searches for run-time libraries and when it resolves undefined symbols, as well as how you can use your own custom functions as a "front-end" to your system's library functions.



* I couldn't think of a better name for it, and it's normally just called "the linker", but that would just be confusing. Oh well.