Compiling Software With Debian Linux

Note: There are many sources available which cover patching and upgrading the kernel. It involves downloading patch files or the Linux kernel source code and compiling a new kernel. This is not the best practice to follow on most Linux systems. Most distributions customize the kernel. Any such customizations would be lost if you do a generic kernel upgrade.

Upgrading the kernel on your Debian system can be done easily (kind of) using the the apt package tool to install a "kernel-image" package. The main reason for upgrading a kernel is to support a new piece of hardware or implement some new function. Note that the Debian CDs offer numerous newer kernel packages. If none of the newer kernels on your CDs don't support some new hardware or function, implement the new hardware driver or function by using a loadable module rather than by upgrading your kernel. Loadable modules are discussed on this page.



There is a lot of free software out there that is written for Linux/UNIX that's only available in source code format. As explained on the Basics page, this is the most efficient way for the developers to distribute their software. When you obtain the source code file(s) you compile them to generate an executable binary that's specific to your OS and hardware platform. Knowing how to compile source files is an indispensible skill in the Linux/UNIX world so we'll dedicate a good part of this page to it.

As mentioned on the Packages page, compiling programs can be a frustrating experience. You can run into all kinds of errors due to outdated library files, certain assumptions made by the programmer (which don't hold true on your system), and numerous other causes. Save yourself some grief and use packages whenever possible or at least use pre-compiled binaries if they're available for your particular system. On the bright side, the fact that Debian is distributed with over 14,000 packages you'd be hard-pressed to find something that isn't available as a package. Having said that, if all you can get is the source code, get that and press on.

There's actually two steps involved in generating an executable binary from a source file, compiling and linking. Most programs, whether written for PCs or mainframes, act pretty much the same way. They'll make "calls" to the operating system for specific "services" like "Give me the current date" or "Tell me which user is running this program". It would be a pain if programmers had to write the same code over and over again in every program they wrote to handle these common functions. To address this there are things called "libraries" that contain the code for tons of these routine tasks. Instead of writing the actual code to do a task, programmers simply add references to these library routines (often referred to as "function calls") in the source code of their program. Each of these routines (or "functions") has a unique name, and the programmer uses this name in their reference.

Once a programmer has written the source code for a program, which you may recall from the Basics page is in a human-readable text file format, they "compile" it using a compiler program. Each programming language has its own compiler (or several of them). The compiler program reads the text source code file as input and spits out a binary file called an "object" file. It's merely a binary (machine-readable) version of the programmer's soure code file, complete with those references to library routines.

The next step is called "linking" and it's where those library references are taken care of. The linker is also just another program. It reads in several files, the object file that was generated by the compiler and one or more library files. Every time the linker finds a reference to a library routine in the object file, it reads the library files and finds that routine. It then replaces the programmer's reference with the code for the routine from the library file. After it has replaced all the references with code from the library files the linker then spits out the executable binary file.

Compiling a program

In Linux, the make utility handles compiling and linking. When you run the make utility you'll see lines that start with gcc which stands for GNU C Compiler. You may also see error messages with the words:

unresolved symbols

This means that the during the linking process a library routine that is referenced in the object file couldn't be found in any of the library files. This may indicate that your library files are out of date or you're missing a library file. Later in this page you'll use the apt utility to add a package which contains a library file because a routine in it is referenced by the source code we use.

Another common reason for errors when trying to compile drivers is that the kernel source code and/or header files are not present. If you plan on compiling drivers for devices (such as the Logitech QuickCam) that don't have a driver included with Debian, you'll need to install those packages before you compile the drivers. (See the Going Modular section below.)

So where does all this compiling and linking take place? Most Linux systems have a /usr/local subdirectory. The "local" refers to things that are specific to the local machine. In other words, if a particular program isn't installed on all Linux systems by default, then it represents something that is additional to that particlar local machine. So most third-party programs and utilities should be installed under this /usr/local subdirectory.

There are quite a few subdirectories under the /usr/local subdirectory. Two of them are /usr/local/src (for source code files) and /usr/local/bin (for binaries). The make utility will usually put any newly-compiled binaries in the /usr/local/bin directory.

With Linux, you typically issue four commands to compile, link, and install a program:

./configure (may not have this)
make
make install
make clean

configure is usually an extremely long and hideously complex shell script that the program's author (hopefully) generated to make your compiling experience easier. It basically checks out your system to see what you have and where it is and to make sure it has everything it needs. I have also seen this same type of script given the name of the program or driver with a .sh extension (example 'myprogram.sh' instead of 'configure'). However, such scripts are no guarantee of a successful compile. Such a script may also create a thing called a "makefile" that is used by the make utility in the next step.

The make utility is what does the actual compiling and linking. If it's going to bomb, this is where it will bomb.

make install puts the compiled binary file in the proper (/usr/local/bin ) subdirectory.

make clean cleans up temporary files that were generated by the compiling and linking processes.

Now that you know what you'll be doing, and how to do, lets compile a program. (Even though compiling is only one step in the process, the term "compile" is often used to refer to the entire process of compiling, linking, and installing.)


Programs and Utilities Top of page

If you're not familier with IRC (Internet Relay Chat), it's like an on-line chat room that you can access using an IRC client. Organizations or individuals will set up IRC servers that those with IRC clients can access and hold on-line, real-time discussions. You can see everything that all other chat members type into their IRC client and everyone else can see everything you type into your IRC client.

Since one of Debian's support options is via an IRC chat room, we'll use the source code for an IRC client for our example. The IRC client we'll use is called ScrollZ (www.scrollz.com). It is a "console" client, meaning it runs in the character-based interface of a shell. No GUI is required. However, it does use a thing called ncurses (also sometimes referred to as just "curses") which allows graphical-type displays on the console screen.

While several ncurses packages are already installed on your system, one is missing. (Found that out through one of those frustrating compiling experiences I was talking about.) It's a library file that contains a routine called tgetent that the linker looks for because it's referenced in the source code. We'll need to add the package that contains this library file. Naturally, your Debian system will need Internet connectivity so you can FTP the ScrollZ tar ball to your system (you'll see how below) or you can download it using a different system and FTP it to your Debian system.

Install the needed ncurses package (off of CD #3) using the command:

apt-get install libncurses5-dev

The next step is to download the ScrollZ client source code tar ball. When you are going to ftp a source code tar ball to your system, you'll want to be in the appropriate subdirectory before you initiate the transfer so that it ends up in the right spot. As such, we use the following two commands to start the process of downloading the ScrollZ source code tar ball:

cd /usr/local/src
ftp ftp.scrollz.com

Enter anonymous when prompted for a user name and your e-mail address when prompted for a password. (This is just customary, they're not collecting e-mail addresses so they can spam you.) Once you're logged in, download the ScrollZ source tar ball using the following commands at the ftp> prompt:

cd pub/ScrollZ/source
ls
get ScrollZ-1.9.3.tar.gz
quit

The ls command will show you what files are available. By the time you try this the ScrollZ-1.9.3.tar.gz may no longer be available so just download the latest version file that ends with the .gz extension. You'll see a line that says:

150 Opening BINARY mode data connection...

and it will just seem to sit there. The file is downloading. There's just no progress indicator.

As mentioned back on the Basics page, when you extract a tar ball it extracts to the same directory structure that was tar-ed. Many tar balls do contain subdirectories. The convention is that the parent directory has the same name as the downloaded file, minus the .tar.gz stuff. (This isn't always followed however.) So if we use the following command to extract the tar ball we just downloaded:

tar -zxvf ScrollZ-1.9.3.tar.gz

we'll end up with the following subdirectory:

/usr/local/src/ScrollZ-1.9.3

which will contain the extracted contents of the tar ball. If we use the commands:

cd ScrollZ-1.9.3
ls -laF

we'll see these contents. You will see several things here that are common practices with source tar balls. You will typically see documentation files with names like (or start with) README which normally contains general information about the program and INSTALL which has instructions for installing the program. (The names being in upper-case is also common for documentation files.) These are text files that you can read with any text editor. There is often also a /doc or /documentation subdirectory with more text file documents like a user manual. If you run into problems installing a program the first place you should look is the documentation files, especially anything related to installation. ScrollZ's INSTALL.ScrollZ installation documentation file is located in the /doc subdirectory.

We see that there is (thankfully) a configure shell script. So as mentioned above, the first command we use is to run this shell script. We precede the name of the shell script with a ./ because the subdirectory it's in isn't in our system's path. So type in:

./configure

and you'll see a ton of text go flying by on your screen. At the end of all of it will hopefully (instead of an error message) be some text telling you to do a 'make'. Just because it tells you do to a make doesn't mean you should. Use the Shift-PageUp key combination to look for lines with "could not find" or "cannot locate" in them. These lines typically list a function name that couldn't be found in any library and you'll need to find out what library is needed before proceeding. If no such lines exist (which they shouldn't because we downloaded the additional library earlier), we can do a make:

make

You may see some warning messages during the compiling process. If any of them are serious enough to be a show-stopper the compile will bomb out with error messages. Otherwise, it will complete successfully with a lot of stuff on the screen.

Next we do a:

make install

to install the files in their respective directories. Then, finally, a:

make clean

Thats it! You've compiled and installed a program from source code. Now you can make sure the program is in the path by typing in:

whereis scrollz

and we see that it's in the:

/usr/local/bin

directory (where the make install put it). Actually, the scrollz file is just a symbolic link to an executable binary named scrollz-1.9.3 which is in the same directory.

Before we run the program we have to set up a few environment variables and files. First we'll add the environment variables to our profile so that we don't have to manually set them every time we use the program. Open your profile file with the command:

nano ~/.profile

(recall that the ~ character represents the path to your home directory) and add the following three statements:

IRCNICK="penguin"
IRCSERVER="irc.debian.org"
export IRCNICK IRCSERVER

Use whatever nickname you want for yourself in place of penguin but make it somewhat unique (and short). Then exit the editor saving the file.

Next, create a directory that the program looks for inside your home directory using the command:

mkdir ~/.ScrollZ

Create an empty 'save' file in this new directory with the command:

touch ~/.ScrollZ/ScrollZ.save

Now, using a text editor, create an empty "rc" file that the program looks for in your home directory (make sure the name starts with a period) with the command:

nano ~/.scrollzrc

and enter the following statement into this file:

set load_path ~/.ScrollZ

Exit the editor saving the file. (Almost done.) Now you can either reboot the system or manually add the above variables to the current running environment using the following commands:

IRCNICK="penguin"
IRCSERVER="irc.debian.org"
export IRCNICK IRCSERVER

That takes care of the program's setup. If you're not still connected to the Internet, reconnect now. Since the program is in our path, all we have to do to run it is type in:

scrollz

You'll see an error saying that it couldn't open the ScrollZ.save file with the cursor flashing in the lower-left corner of the screen. Now that you're in the program, you'll be entering IRC (and some ScrollZ) commands, not Linux commands. Enter the following command:

/sve

and press Enter to create the file and save the settings.

You're now connected to debian.org's IRC server but you have to enter the Debian room. To do that, enter the IRC command:

/join #debian

You're in! Anything you type on the line below the blue bar (where the cursor is currently flashing) will be displayed in the room (after you press Enter). That's why you have to start all IRC commands with a slash (/) character. Those aren't displayed as chat. The lj on the right side of the blue bar stands for "last joined" and your nickname will be there until someone else joins the room after you.

So what if you're in the chat room and you want to try something at a shell prompt? Just press the Alt-F2 key combination to get to a different console. Pressing Alt-F1 will take you back to the console your chat session is in.

The following IRC commands will be helpful during your chat sessions:

/join #room-name
/leave #room-name
/quit parting message to participants
/msg individual-participant-nickname private-message
/whois your-nickname
/whois another-participants-nickname

Note that /quit exits the ScrollZ program in addition to leaving the room.

When the room is busy it's best to begin messages intended for a specific particpant with that participant's nickname. When asking for help with a problem, be as specific as possible and supply as much relevant information as possible.

That's the good news. Now the bad news. In many cases the above compiling/linking/installing procedure won't do you any good. Many source tar balls don't contain a configure shell script. In that case all you can do is look for documentation files or Web pages on the author's Web site for compiling and installation information. In some cases the author may assume those using their program are experienced Linux/UNIX administrators and won't offer much in the way of documentation. If this is the case, e-mail them asking for more specific instructions. Their e-mail address or the URL to their Web site will likely be given in one of documentation files.

Drivers

That's compiling a program. The other big thing that often needs compiling are drivers. Most drivers compile into driver "modules" which we explain in the next section.

Let's look at the driver for the popular 3Com 3C509 NIC. The source code for the driver is available at www.scyld.com/network/3c509.html. Near the bottom of the source code file are the following commented lines:

/*
* Local variables:
* compile-command: "gcc -DMODULE -D__KERNEL__ -Wall -Wstrict-prototypes -O6 -c 3c509.c"
* version-control: t
* kept-new-versions: 5
* c-indent-level: 4
* c-basic-offset: 4
* tab-width: 4
* End:
*/


Notice that the command needed to compile the driver is given in these comments. However, also on the above Web page, right next to the link for the driver source code, is a link on how to set the driver up as a module. Any author or company that makes driver source code available should supply you with similar information either as comments in the source code, in an accompanying README file, or on a Web page. If they don't, e-mail them asking for it. If they get enough e-mails perhaps they'll find it beneficial to create a README file or Web page.


Going Modular Top of page

One of the things the kernel has to do is interface with certain pieces of hardware. Like other operating systems, these pieces of hardware have their own individual software drivers. One of the reasons the kernel has grown in size over the years is that more and more drivers for more and more hardware devices have been compiled into the kernel.

With all of the new hardware coming out, trying to compile in all of the available hardware drivers would result in a huge kernel that takes up a good chunk (if not all) of your system's memory. While it may make sense to compile certain drivers into the kernel (like IDE and SCSI hard-drive controller drivers), the majority of hardware drivers can be compiled as loadable modules and dynamically added to the running kernel as needed.

In addition to hardware device drivers, modules are also developed for new filesystems and new kernel routines like data encryption. Adding modules does not increase the size of the kernel. They merely take up more disk space (and more memory if they are loaded).

By modulizing drivers you reduce the size of the base kernel which cuts down kernel memory requirements. Using modules is also safer. If you compile a new driver into a kernel there's no way to get it out. If there's a bug in it you're pretty much screwed. You'd have to roll back to your previous kernel. Being able to manage loadable modules means that if you load a new module and it causes problems, you simply unload it.

The nice thing about modules is that if you do find yourself needing to add some new drivers or new functionality to your kernel you can do it without patching or replacing the kernel source code and recompiling everything. Loadable modules are like adding code to a running kernel.

These days drivers are all written loadable modules and a lot of times a driver module is only available in source code format (sourceforge.net provides access to many of these). Compiling a driver module isn't much different than compiling a program. You'll just end up with a file with a .o extension (the letter 'O' - not zero).

Note: The source code for many driver modules requires access to the kernel source code or just the kernel headers during the compilation process. You are not recompiling the kernel. It's just that the driver needs access to certain kernel header, source, etc. files.

If unsure what is needed by a specific driver, try compiling it with just the header files first. Debian includes a lot of "kernel-header" packages on the binary CDs. Enter the command:

apt-cache search kernel-headers

to see all of them. It's IMPORTANT that you install the one that matches the kernel you're running. Enter the command:

uname -a

to find out what kernel version you're running and install the header package that matches it exactly (it may differ depending on the processor in your system). On my system it was:

apt-get install kernel-headers-2.4.27-1-386

If the kernel source code is needed you can get that off of the binary CD set as well. Note that the kernel source tar ball is about 180 meg in size so you'll want to use the df -h command to make sure you have about 400 meg free on your hard-drive before you install the kernel source. Use the command:

apt-get install kernel-source-2.4.27

to load the kernel source off the Debian binary CD set onto your system. Note that this installs a single BZIP (compressed) file of the source code into the /usr/srcdirectory. You have to first unbzip it with the command:

bunzip2 kernel-source-2.4.27.tar.bz2

(It'll take awhile.) Then you have to untar it to create the kernel source tree using the command:

tar -xvf kernel-source-2.4.27.tar

You can delete the .tar file after the extraction to free up some disk space.

The utilities you use to manage modules are contained in the modutils package. These utilities include:

* lsmod - lists currently loaded modules
* insmod - inserts modules into the kernel
* rmmod - removes modules from the kernel
* modprobe - a better alternative to insmod because it checks for dependent modules
* depmod - builds a file which contains the dependency information used by modprobe


There are other utilities but these are the most commonly-used. They are all executable binary files and all are located in the /sbin directory.

An important point to remember is that modules may be specific to a kernel version or at least a given kernel version and higher. If a module wasn't written for your kernel version you'll get an error when you try and load it. Module file names end with a .o and are located in a subdirectory that has a parent directory with a name that includes the Linux kernel version. For example, the module files on a Debian 3.1 system are in subdirectories similar to:

/lib/modules/2.4.27-1-386/kernel/drivers/ide
/lib/modules/2.4.27-1-386/kernel/drivers/net
/lib/modules/2.4.27-1-386/kernel/drivers/video

In addition,

/lib/modules/2.4.27-1-386

is also where the file modules.dep file is located. The modules.dep file is where the module dependecy information generated by the depmod utility is stored and accessed by the modprobe utility.

For the curious, if you go looking through the system's startup files looking for insmod or modprobe commands you won't find any. One of the scripts in the /etc/init.d subdirectory that's called pretty early in the sartup process is the kerneld shell script. The kerneld shell script calls the /sbin/kerneld executable binary that causes modules to be auto-loaded as they're needed. In addition, optional modules listed in the /etc/modules file are also loaded.

If you boot your system and use the Shift-PageUp key combination to look at the messages generated early in the boot process, you'll see modules start to get loaded shortly after the init process starts.

It's at system startup where you're most likely to see module-related messages and errors. The most common one starts out:

insmod: /etc/modules.conf is more recent than....

This isn't really a problem, just a file date issue. Most of the time it'll just disappear in the course of using your system.

Compiling A Driver Module

Back on the Webcam page we found that Debian doesn't come with a driver for the popular QuickCam USB camera. There is a driver for it on sourceforge.net called qc-usb. This driver comes with a script that checks for certain things and contains the necessary compile commands. Before proceeding Make sure you've installed the kernel header files as explained earlier.

You can download the driver using the FTP utility on your Debian system (as we showed above for the ScrollZ program file) or download it using another system and FTP it to your Debian system. If you do the later you probably FTPed it up to your user home directory so you'll need to copy the file to the proper directory with the command:

cp /home/keith/qc-usb-0.6.2.tar.gz /usr/local/src

Then go into the /usr/local/src/ subdirectory and enter the command:

tar -zxvf qc-usb-0.6.2.tar.gz

and a /usr/local/src/qc-usb-0.6.2 subdirectory will be created and contain all of the files. If you list the files in the directory you'll see the quickcam.sh file which is the setup script written by the driver developer in an attempt to make creating the driver module easier for you.

Be sure you have the cam plugged in at this point. We need to supply to the quickcam.sh script with the location of the kernel header files we installed earlier. To run the script and supply it with this information we use:

./quickcam.sh LINUX_DIR=/usr/src/kernel-headers-2.4.27-1-386

It'll start out by checking for files and utilities. Disregard the warning about exporting kgcc. It'll then warn you that you shouldn't run as root but we're going to do it anyway since later in the script it would ask for the root password if we weren't.

At one point you'll see a bunch of insmod error messages. It's just trying to find out what type of USB circuitry you have and they can be ignored. Eventually you'll see a bunch of gcc compiler commands and when it's done the script will try and load the driver. When you see the line:

Name : Logitech QuickCam USB

several lines above the prompt to press Enter to continue, press Ctrl-C to exit the script. This is because we're not running a GUI. The script didn't do one thing. It didn't copy the driver to the proper location to have it auto-load during system bootup (if the cam is connected to the system when it boots up). To do that, use the command:

cp quickcam.o /lib/modules/2.4.27-1-386/kernel/drivers/usb

That's it! Reboot the system and enter the lsmod command at the shell prompt and you should see the quickcam driver loaded. Now that you have the driver auto-loading you can return to the Webcam page to continue setting up your cam server.

Unfortunately not all driver modules include setup scripts but at the very list there should be documentation available that tells you how to compile it. (Be sure to put the final .o driver file into the correct subdirectory under the /lib/modules/2.4.27-1-386/kernel/drivers directory.) The one downside to a script is that the person that wrote it may make certain assumptions about file locations, etc. that aren't true and the script may bomb out. If a script does bomb don't assume the driver won't work on your system. Try using the documentation to compile the driver manually before you give up.