How to run proprietary programs from different versions of a Linux installation in a compatibility environment

Novell Cool Solutions Feature
By Manfred Hollstein

Posted: 11 Oct 2004

White Paper written by Manfred Hollstein
SUSE LINUX AG, a Novell company

The purpose of this document is to provide a migration path for users who have invested in proprietary application packages, which were certified for specific versions of various basic libraries such as glibc only, but intend to run them on different versions now.

The reader of this document should be able install and configure the files required for such a task. In addition, one will find hints in case something goes wrong.

This document is a result of using the described steps in a real project.

	Background

A Linux program usually consists of machine code to perform a specific set of functions. Normally, some of these functions are provided by libraries; such libraries can exist as a static version included in an archive, or they can exist as a dynamic, shared object. When a program is created, the developer decides whether static or shared libraries will be used. There are only a few instances when a static library should be used; such instances include binary incompatibilities between different versions of a library, or evolving changes in the infrastructure of such a library. A typical example of the latter is what occurred while the programming language C++ matured, but still wasn't commonly standardized; implementation details like how virtual functions in derived inheritance lattices are addressed resulted in different, incompatible versions of the same library dependent on the version of the compiler that was used to create the library. A shared library has several benefits: its machine code is only loaded once into the computer's memory, it's shared between several programs using this shared library; and only the memory for program-specific data needs to be allocated. Another benefit is that a system administrator can easily upgrade a shared library without having to touch any programs using that library; this is especially important for security fixes.

Everything which has been said above is not only Linux-specific, but applies to other modern operating systems which support shared libraries as well.

	The Scenario

Do you remember the good old days when you wanted to execute a newly downloaded program on your brand new, leading-edge Linux system, only to find that it didn't work? Most often this happened during the transition phase from libc4 to libc5, or from libc5 to libc6 which is also known as glibc (the GNU C library). The C library is probably the most important library on every UNIX/Linux system, because it acts as the interface between each application and the kernel. Changes in such a central library can easily render a system useless, if such changes are not backward compatible.

There are other scenarios as well, such as when programs were built elsewhere on a system which had newer versions of those libraries installed than you have on your own. It is quite likely that such programs will not even start or work properly on your system; this happens if the program is actively making use of the new/changed features in the newer versions of the libraries.

It is common sense that versions of a library with functionality which was not included in former versions, carry a version number indicating that programs linked to an older version cannot be loaded and executed. This version number is called the major revision of a shared library. The following diagram explains this by using glibc as an example:

GLIBC version	Filename
libc5	/lib/libc.so.5
libc6	/lib/libc.so.6

To maintain certain minor revisions of a shared library, the filename can be augmented by additional suffixes such as libc.so.6.3.3.

As has been shown in the Backgrounds section above, there are further complications to the scenario so far; this is caused by the described binary incompatibilities of C++ libraries compiled with GNU C++ compilers prior to v3.3. Even with a currently standardized environment including consistent API and ABI definitions, it is still not yet clear if future versions of C++ compilers and libraries will remain 100% compatible. To address such issues, the Linux Standards Base (LSB) recommended that every software vendor creating software written in C++ should statically link the C++ parts. Unfortunately, very few vendors actually followed this recommendation. To make things worse, there is a Linux distributor who created their own version of a C++ compiler, but failed to ensure that their shared libraries carried a different version than those official versions.

In short, if you want to start a program and it fails to actually start by emitting error messages similar to the following, your problem can most likely be solved pretty easily by following the suggestions described in this article.

Error message when starting a C program

./a.out: relocation error: ./a.out: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

If you happen to see the above error message about the symbol errno, your program has been linked with a version of glibc older than 2.3, because newer versions of glibc no longer provide errno as a global variable, to allow its thread-safe usage. In this situation you would have to setup a compatibility environment with an older version of glibc such as version 2.2.5.

If, however, you don't see any error message at all, and the program starts but doesn't work as you'd expect it to, you might be hitting the C++ problem described above. The runtime linker, which is loading your program into memory and resolving any dependencies with shared libraries, is able to locate the required shared libraries on your system and to load them into memory, but they may not fit with the actual program.

	Diagnosing problems

The first thing to identify problem sources is to run the command ldd on your programs. The purpose of this program is to print the shared libraries required by each program or shared library specified on the command line. Running ldd on /bin/bash will show output similar to the following:

ldd /bin/bash
        linux-gate.so.1 =>  (0xffffe000)
        libreadline.so.4 => /lib/libreadline.so.4 (0x40036000)
        libhistory.so.4 => /lib/libhistory.so.4 (0x40062000)
        libncurses.so.5 => /lib/libncurses.so.5 (0x40069000)
        libdl.so.2 => /lib/libdl.so.2 (0x400af000)
        libc.so.6 => /lib/tls/libc.so.6 (0x400b2000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
If a shared library cannot be found, an error message will be shown:
        libfoo.so.1 => not found

This error message tells you that the program you're trying to execute actually needs a shared library libfoo.so.1, but cannot find it in the predefined locations that shared libraries are looked for. These predefined locations normally contain the directories /usr/lib and /lib; in addition, a system administrator can add additional directories to a file /etc/ld.so.conf to instruct the runtime linker where to search for shared libraries. As a last resort, an environment variable LD_LIBRARY_PATH can be defined containing a list of additional directories; on Linux, this variable will not be used for programs which run under Ids which do not belong to the user.

	The Solution

Once you have identified the source of the problem, you can start to setup a compatibility environment to provide the required functionality to execute your program properly. In the following example we use the following terminology:

Program name to be migrated	zoo
Prefix ($prefix) directory location to install the compatibility files	/opt/compat-env/zoo
List of required libraries	glibc-2.3.2-95.27 libgcc-3.2.3-42 libstdc++-3.2.3-42
Source ($srcdir) directory where required files are located initially	/root/zoo-src

Step 1

Create directories bin and lib under the prefix directory:

% prefix=/opt/compat-env/zoo
% mkdir -p $prefix/bin $prefix/lib

Step 2

Place copies of the required files into their respective locations:

% srcdir=/root/zoo-src
% cd $prefix/bin
% cp -p $srcdir/zoo .
% cd $prefix/lib
% rpm2cpio $srcdir/glibc-2.3.2-95.27*.rpm | cpio -idvm '*libc*.so*'
% find . -name '*libc*.so*' -exec mv -v '{}' . \;
% rpm2cpio $srcdir/glibc-2.3.2-95.27*.rpm | cpio -idvm '*ld*.so*'
% find . -name '*ld*.so*' -exec mv -v '{}' . \;
% rpm2cpio $srcdir/libgcc-3.2.3-42*.rpm | cpio -idvm '*libgcc_so*.so*'
% find . -name '*libgcc_so*.so*' -exec mv -v '{}' . \;
% rpm2cpio $srcdir/libstdc++-3.2.3-42*.rpm | cpio -idvm '*libstdc*.so*'
% find . -name '*libstdc*.so*' -exec mv -v '{}' . \;
% rm -rf lib usr

Step 3

Create a wrapper script to setup the appropriate environment variables and start the program:

% cd $prefix/bin
% mv zoo zoo.exec
% cat > zoo << EOF
#! /bin/bash
prefix=/opt/compat-env/zoo
LD_LIBRARY_PATH=”$prefix/lib${LD_LIBRARY_PATH+:${LD_LIBRARY_PATH}}”
export LD_LIBRARY_PATH
$prefix/lib/ld-linux.so.2 ${0}.exec “$@”
EOF
% chmod 755 zoo

Now the program zoo can be executed, and the versions of glibc, libgcc and libstdc++ will be used as per requirement.

Still have problems?

If you have setup everything as described, it is still possible that the program cannot be executed; instead an error messages like the following is printed:

symbol __libc_wait, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

Newer versions of the GNU C library assign version numbers not only to the library itself as a whole, but to individual symbols as well. This usually happens to conform with certain language standards; in the example, the global symbol __libc_wait with version number GLIBC_2.0 isn't defined anymore. This could be solved by using an older version of the C library, or one can use another nifty feature of glibc and its runtime linker. It is possible to preload shared objects to override symbols defined in shared objects which would be loaded in the course of the runtime linker's normal dependency resolution. This can also be used to define symbols which are otherwise not defined, such as the __libc_wait function in the example.

The proper fix for this problem would be to create a small shared object containing the required implementation, and to load it in the wrapper script:

% cd $prefix/lib
% cat > libcwait.c << EOF
#include <errno.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <sys/wait.h>
pid_t __libc_wait (int *status)
{
  int res;
  asm volatile ("pushl %%ebx\n\t"
                "movl %2, %%ebx\n\t"
                "movl %1, %%eax\n\t"
                "int $0x80\n\t"
                "popl %%ebx"
                : "=a" (res)
                : "i" (__NR_wait4), "0" (WAIT_ANY), "c" (status), "d" (0),
                  "S" (0));
  return res;
}
EOF
% gcc -fpic -O2 -shared libcwait.c -o libcwait.so

Now make the required modifications to the wrapper script:

% cd $prefix/bin
% cat > zoo << EOF
#! /bin/bash
prefix=/opt/compat-env/zoo
LD_LIBRARY_PATH=”$prefix/lib${LD_LIBRARY_PATH+:${LD_LIBRARY_PATH}}”
export LD_LIBRARY_PATH
LD_PRELOAD=$prefix/lib/libcwait.so
export LD_PRELOAD
$prefix/lib/ld-linux.so.2 ${0}.exec “$@”
EOF
% chmod 755 zoo

	Conclusion

These hints should suffice to create a compatible environment, allowing a user to execute programs which their vendor has not (yet) ported or built on a specific version of a Linux operating system. It can be easily used to setup everything to run programs from a Red Hat Enterprise system on a SUSE LINUX Enterprise Server and vice versa.

Additional information can be found in the GNU C library documentation, which should be included with your particular distribution, and at the LSB website http://www.linuxbase.org/.

Reader Comments

• Wow, this hit the spot.
• Found this very helpful.
• Exactly what I needed to know