Fun with strace and the GDB Debugger

The UNIX family has always provided abundantly for its users. UNIX is a treasure chest oftools with which you can not only do productive work but also educate and entertain yourselfas you explore the depths of the operating system. Two useful tools for this purpose are strace,with which you can trace the system calls of any program, and the GDB Debugger, whichis a full-featured debugger that allows you to run programs in a controlled environment.

The UNIX design consists of hundreds of function calls (called system calls) coveringsimple tasks, such as displaying a character string on the screen to setting task priorities. All UNIX programs accomplish their tasks by calling these low-level services thatthe operating system provides, and with the strace tool, you can actually see these calls andthe parameters that they use. In this way, you can actually play with programs to learn abouttheir low-level interactions with the operating system.

Let the games begin

Let's begin by examining a simple UNIX command -- pwd -- andthen dive deeper into what the command does to accomplish its purpose. Launch an xterm tocreate a controlled environment to experiment with, and then type the command:

$ pwd

The pwd command displays the current working directory. The outputon my computer at that moment in time was:

/home/bill/

Such a simple function belies the complexity beneath the surface of the command (as do allcomputer programs, by the way). To get a picture of this complexity, run thepwdcommand again using the strace tool:

$ strace pwd

With that command, you can see just how much goes on in the UNIX machine to discover andlist the current working directory you're in (seeListing 1).

Listing 1: Output from the strace pwd command

                execve("/bin/pwd", ["pwd"], [/* 39 vars */]) = 0uname({sys="Linux", node="sammy", ...}) = 0brk(0)                                  = 0x804c000old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001......fstat64(3, {st_mode=S_IFREG|0644, st_size=115031, ...}) = 0old_mmap(NULL, 115031, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40017000close(3)                                = 0open("/lib/tls/libc.so.6", O_RDONLY)    = 3read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360U\1"..., 1024) = 1024fstat64(3, {st_mode=S_IFREG|0755, st_size=1547996, ...}) = 0old_mmap(0x42000000, 1257224, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x42000000mprotect(0x4212e000, 20232, PROT_NONE)  = 0old_mmap(0x4212e000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x12e000)...old_mmap(0x42131000, 7944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS,...close(3)                                = 0set_thread_area({entry_number:-1 -> 6, base_addr:0x40016ac0, limit:1048575, seg_32bit...munmap(0x40017000, 115031)              = 0brk(0)                                  = 0x804c000brk(0x804d000)                          = 0x804d000brk(0)                                  = 0x804d000open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3fstat64(3, {st_mode=S_IFREG|0644, st_size=30301680, ...}) = 0mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40017000close(3)                                = 0brk(0)                                  = 0x804d000brk(0x804e000)                          = 0x804e000getcwd("/home/bill", 4096)              = 11fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 6), ...}) = 0mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4021700...write(1, "/home/bill\n", 11/home/bill)            = 11munmap(0x40217000, 4096)                = 0exit_group(0)                           = ?

---

Back to top

The nuts and bolts of UNIX system calls

It is beyond the scope of this article to go into too much detail about why all these systemcalls are necessary for retrieving and displaying the current working directory, but I will show how to obtain that information. In each line ofListing 1, a system call and its parameters are spelled out in aC-like format, just as aC programmerwould expect to see them. Strace displays the calls like this as well, regardless of the actualprogramming language used in the creation of the program.

If you want to understand all the details that appear in Listing 1, UNIX provides massive amountsof documentation on all the system calls. If there's a "most important" function inListing 1, it isthe getcwd() function, which stands for get current workingdirectory. With the current xterm still showing the output ofstrace pwd,launch another xterm and type the following command to see what UNIX displays for this function:

$ man getcwd

What you should see is a complete listing of the getcwd() functionand a listing of the arguments that this importantC function requiresand returns. Likewise, you can typeman brk orman fstat64, and so on. These system functions are typically welldocumented, and if you take the time to study them, you will begin to understand just how powerfulUNIX is and how easy it is to perform a detailed study of the low-level system. Of all operating systems,UNIX is best prepared to help you understand what goes on within it out of the box.

---

Back to top

Discover nweb

For the next steps, you need something larger and more complex than a simple UNIX commandlikepwd. A simple Hypertext Transfer Protocol (HTTP) server, such asnweb, is perfect. An HTTP server listens for your browser requests when you'resurfing the Internet,and then responds to your browser requests by sending the objects you're requesting, such as Webpages and graphics files.

Download and install nweb, written by IBM developerWorks contributing writer, Nigel Griffiths. (SeetheResources section for a link to Nigel's article, "nweb: a tiny, safeWeb server (static pages only)" (developerWorks, June 2004).)

After downloading es-nweb.zip to your $HOME/downloads directory, type the simple commandsshown inListing 2 to extract, compile, and launch the program:

Note: I assume that you're going to compile this program to a Linux®workstation. If this is not the case, read the nweb article for details on compiling theprogram on other UNIX operating systems.

Listing 2. Commands for extracting, compiling, and launching nweb

                $ cd src$ mkdir nweb$ cd nweb$ unzip $HOME/downloads/es-nweb.zip$ gcc -ggdb -O -DLINUX nweb.c -o nweb$ ./nweb 9090 $HOME/src/nweb &

Note: The -ggdb option in Listing 2 differs from Nigel'sarticle in that it tells the GCC compiler to optimize the program for debugging with the GDBDebugger, which you'll use later.

Next, to verify that the nweb server is running, use the ps commandshown inListing 3 to check it.

Listing 3. The ps command

                $ ps  PID TTY          TIME CMD 2913 pts/5    00:00:00 bash 4009 pts/5    00:00:00 nweb 4011 pts/5    00:00:00 ps

Finally, to verify that nweb is really running and that all is correct, launch a Web browseron your computer and typehttp://localhost:9090 in the addressbar.

Using strace with nweb

Now, let's have some fun. Launch another xterm, and then use strace to trace thenweb server that is running. To do so, you must know the process ID of the program,and you must have the appropriate permissions. You'll watch only a specific set of systemcalls -- those related to networking. Begin by typing the command shown in the first line ofListing 4 (using the nweb process ID displayed above). You should see the output below(line 2 in Listing 4).

Listing 4. Starting a trace of nweb

                $ strace -e trace=network -p 4009accept(0,

Notice that the trace has stopped in the middle of a call to the network accept()function. Refresh the http://localhost:9090 page in yourbrowser a few times, and notice what strace displays each time you refresh the page.Isn't that great? What you are watching is a low-level network call by the HTTP server,nweb, when it has been called by your Web browser. Simply put,nweb is accepting your browser's call.

You can stop tracing the network calls on the running nweb process by pressing Ctrl+Cwhen the xterm in which the strace is running has the window focus.

---

Back to top

On to the GDB Debugger

As you saw, strace can be a great program for learning how user programs interact with theoperating system through certain system calls. The GDB Debugger can also attach itself to acurrently running process and help you to dig even deeper.

The GDB Debugger is such a useful tool that a lot of information about it is available on theInternet. Debuggers in general are valuable tools, and anyone responsible for the developmentand upkeep of a computer system should know how to use them. So, while nweb is still runningin another xterm session, halt the strace by pressing Ctrl+C, and then launch the GDB Debuggerby typing the commands shown inListing 5.

Listing 5. Launch the GDB Debugger

                $ gdb --quiet(gdb) attach 4009Attaching to process 4009Reading symbols from /home/bill/src/nweb/nweb...done.Reading symbols from /lib/tls/libc.so.6...done.Loaded symbols for /lib/tls/libc.so.6Reading symbols from /lib/ld-linux.so.2...done.Loaded symbols for /lib/ld-linux.so.20xffffe410 in ?? ()(gdb)

The -quiet option tells the GDB Debugger to display only its promptand not all the other startup information that it typically displays. If you want the extra text, leave the-quietoption off.

The attach 4009 command starts debugging the currently runningnweb server, and the GDB Debugger responds in kind by reading all the symbolic information aboutthe process it can. Next, use theinfo command to list informationabout the program you're exploring (seeListing 6).

Listing 6. The info command lists program information

                (gdb) info procprocess 4009cmdline = './nweb'cwd = '/home/bill/src/nweb'exe = '/home/bill/src/nweb/nweb'(gdb)

Another useful variant of the info command (see Listing 7) isinfo functions; however, the list of functions can be very long.

Listing 7. Functions list from the info functions command

                (gdb) info functionsAll defined functions:File nweb.c:void log(int, char *, char *, int);int main(int, char **);void web(int, int);File __finite:int __finite();...(gdb)

Because you compiled the nweb program with the -ggdb option, alot of debugging information was included in the executable, allowing the debugger to see the definedfunctions listed by file, as shown inListing 7.

---

Back to top

The list and disassemble commands

Two important GDB Debugger commands are list anddisassemble. Try these commands by working with the codeinListing 8.

Listing 8. The list command

                (gdb) list main121             exit(1);122     }123124125     main(int argc, char **argv)126     {127             int i, port, pid, listenfd, socketfd, hit;128             size_t length;129             char *str;130             static struct sockaddr_in cli_addr; /* static = initialised to zeros */(gdb) 131             static struct sockaddr_in serv_addr; /* static = initialised to zeros */132133             if( argc < 3  || argc > 3 || !strcmp(argv[1], "-?") ) {134                     (void)printf("hint: nweb Port-Number Top-Directory\n\n"135             "\tnweb is a small and very safe mini web server\n"136             "\tnweb only servers out file/web pages with extensions named below\n"137             "\t and only from the named directory or its sub-directories.\n"138             "\tThere is no fancy features = safe and secure.\n\n"139             "\tExample: nweb 8181 /home/nwebdir &\n\n"140             "\tOnly Supports:");

As you can see, the list command lists the running program insource form with line numbers. Pressing the Return key (shown between lines 130 and 131) simplycontinues listing from where the last list left off. Now, try thedisassemblecommand, which you can abbreviate to disass (seeListing 9).

Listing 9. The disassemble command

                (gdb) disass mainDump of assembler code for function main:0x08048ba2 <main+0>:    push   ebp0x08048ba3 <main+1>:    mov    ebp,esp0x08048ba5 <main+3>:    push   edi0x08048ba6 <main+4>:    push   esi0x08048ba7 <main+5>:    push   ebx0x08048ba8 <main+6>:    sub    esp,0xc0x08048bab <main+9>:    mov    ebx,DWORD PTR [ebp+12]0x08048bae <main+12>:   and    esp,0xfffffff0...0x08048c01 <main+95>:   call   0x8048664 <printf>0x08048c06 <main+100>:  add    esp,0x100x08048c09 <main+103>:  inc    esi0x08048c0a <main+104>:  cmp    DWORD PTR [ebx+esi],0x0---Type <return> to continue, or q <return> to quit---

This disassembly listing shows the assembler-language listing of the mainfunction. In this case, the assembler code indicates that the computer running the codehas an Intel® Pentium® processor. Your code will look quitedifferent if it's running on a computer with a different processor type, such as anIBM Power PC®-based computer.

---

Back to top

Watching it live

Because you're watching an actual running program, you can set breakpoints, and thenseethe program as it replies to browser requests and transmits .html and .jpg files to the browsermaking the request.Listing 10 shows how you can do so.

Listing 10. Set breakpoints

                (gdb) break 188Breakpoint 1 at 0x8048e70: file nweb.c, line 188.(gdb) commands 1Type commands for when breakpoint 1 is hit, one per line.End with a line saying just "end".>continue>end(gdb) cContinuing.

At this point, the GDB Debugger is set to break at the line where the nweb serveracceptsbrowser requests; the debugger will simply display the request and continue processing otherrequests without interrupting the running program. Refresh thehttp://localhost:9090/page in your browser a few times, and watch the GDB Debugger display the breakpoint andcontinue running.

While refreshing the browser page, you should see breakpoint information, such as that showninListing 11, scrolling in the GDB Debugger xterm. Just as with strace, you can stop debuggingthe nweb server by pressing Ctrl+C. After stopping the tracing, you can exit the GDB Debuggerby typing thequit command.

Listing 11. Breakpoint information in the GDB Debugger xterm

                Breakpoint 1, main (argc=3, argv=0x1) at nweb.c:188188 if((socketfd = accept(listenfd, (struct sockaddr *)&cli_addr, &length)) < 0)Breakpoint 1, main (argc=3, argv=0x1) at nweb.c:188188 if((socketfd = accept(listenfd, (struct sockaddr *)&cli_addr, &length)) < 0)Breakpoint 1, main (argc=3, argv=0x1) at nweb.c:188188 if((socketfd = accept(listenfd, (struct sockaddr *)&cli_addr, &length)) < 0)Breakpoint 1, main (argc=3, argv=0x1) at nweb.c:188188 if((socketfd = accept(listenfd, (struct sockaddr *)&cli_addr, &length)) < 0)Program received signal SIGINT, Interrupt.0xffffe410 in ?? ()(gdb) quitThe program is running.  Quit anyway (and detach it)? (y or n) yDetaching from program: /home/bill/src/nweb/nweb, process 4009$

Notice that you're telling the GDB Debugger to quit debugging a program that is still active inmemory. Even after leaving the debugger, you can refresh the browser page and see thatnweb is still running. You can halt the program by typing thekill 4009command, or the page will disappear when you leave your session.

As always, you can learn a lot about tools, such as strace and the GDB Debugger, from theirman and info pages. Be sure to use the tools that UNIX provides you!

---

Back to top

Learn as much as you can

It's never a bad thing to learn as much as you can about the computer that you're working on,nor is it bad to have fun in the process. UNIX actually encourages you to explore and learn byproviding tools, such as strace and the GDB Debugger, as well as a wealth of information in theman and info pages. Computers are an extension of our minds, and the more we know aboutthem, the more useful they become.

Resources

Learn

• AIX and UNIX articles: Check out other articles written by William Zimmerly.
  
  
• "nweb: a tiny, safeWeb server (static pages only)" (developerWorks, June 2004): Get the source code for the nweb HTTP server.
  
  
• "Mastering Linux debuggingtechniques" (developerWorks, August 2002):Steve Best of IBM discusses key strategies for locating and stomping bugs in Linux systemsand describes using strace, the GDB Debugger, and several other useful command-line tools.
  
  
• "Debugging toolsand techniques for Linux on Power" (developerWorks, August 2005): Calvin Sze, a Linux consultant for IBM, discusses strace, the GDB Debugger, and graphical debuggers.
  
  
• UNIX man pages: strace (1M): Viewthe UNIX man page strace documentation from a University of Alabama documentation server.
  
  
• GDB: Check out the GDB man page fromLinuxCommand Web site.
  
  
• The xterm command: Learn more about xterm from the IBM online public library and information center.
  
  
• Technical events and webcasts: Stay current with developerWorks technical events andwebcasts.
  
  
• AIX and UNIX:Want more? The developerWorks AIX and UNIX zone hosts hundreds ofinformative articles and introductory, intermediate, and advancedtutorials.