Remote Kernel Debugging with WinDbg

original url from : http://www.wd-3.com/archive/RemoteDbg.htm

Remote Kernel Debugging with WinDbg
March 15,2003
JolyonWright

Copyright © 2003 by Jolyon Wright. All rights reserved

The standard way to use Microsoft's WinDbg debugger is toconnect two PC's with a null modem cable. This configuration is fine for theinitial development of a driver. As soon as you unleash your code on theunsuspecting world, you need a slightly more sophisticated model. Thedocumentation can be daunting because it covers every possible permutation ofdebugger and transport protocol. WinDbg has evolved at such breakneck speed thatthe documentation has struggled to keep up. The Debugger has a multitude ofdifferent users, with different needs and different backgrounds.

In this article, I will illustrate a straightforward and extensible approachto integrating an additional Remote Host computer using a network connection. Iwill also demonstrate how this simple model can be extended to produce debuggerconfigurations of dazzling complexity. Using the simple steps I'll outline hereyou can diagnose machine specific problems from the tranquillity of your officeand have multiple developers simultaneously poring over the same debuggingsession. All of the material presented here is covered in the WinDbgdocumentation, but I found it challenging extrapolating the relevant nuggets ofinformation. I aim to alleviate this chore for other developers.

Review of the Standard Target and Host Setup

The WinDbg documentation does a good job of describing the standard kerneldebugging setup, but it is worth reviewing briefly. The simplest configurationfor kernel debugging uses two computers: a Target Machine that runs the driverunder test and a Host Machine that runs the debugger.  Figure 1 depictsthis configuration:-

 

Figure 1. Target and Host.

I am not going to explore this configuration in any detail. You normallyplace the source for your driver on the Host Machine, which is also yourpersonal development machine. In addition, the driver symbols (a byproduct ofthe build process) will probably be on the Host as well. This simpleconfiguration has the benefit of simplicity, but it severely limits our optionsfor remoting the debug session. In an environment where multiple developers areworking on a project, we need to ensure that both the Source and the Symbols arein a central place rather than on a development machine that isn't widelyaccessible. Centralizing source and symbols opens up a load of debuggingpossibilities.

Getting the Symbols and Source off the Host

In this scenario we are going to clear our Source and Symbols off the HostMachine and archive all of it off onto a Server. I am going to describe how toconfigure the Target and Host at a genteel pace, before quickly checking that itworks. This setup forms the basis for the more esoteric configurations in thefollowing sections. Figure 2 shows the Host, Target and Server.

Figure 2. Target, Host and Server.

The Target Machine has been setup with a shrink wrap XP installation. Theonly "tweak" is a modifiedboot.ini file. The boot.ini filecontains a list of the alternative operating systems that you can boot from themain partition. Right after you install Windows XP, there will be just one entryin this file. We want to modifyboot.ini to enable us to hookup adebugger to the serial port. The easiest way to edit this file is to open acommand prompt, change the attributes for the file, and launch Notepad:

C:\>attrib -s -h -rboot.ini
C:\>notepad boot.ini

 

Now edit the boot settings to add /debugport and/baudrateswitches in the [operating systems] section.

[boot loader]timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operatingsystems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XPProfessional" /fastdetect/debugport=COM1 /baudrate=115200

The next timeyou boot the target machine, the new switches will cause the kernel to waituntil it can attach to WinDbg before proceeding.

In this configuration, you install WinDbg on the host machine. I am usingversion 6.1.0.0017.2, but this will undoubtedly be out of date before I reachthe end of this paragraph. We need to set some environment variables to ensurethat the debugger can resolve both the operating system symbols and the symbolsfor my drivers. We also need to ensure that the Host's baudrate matches the baudrate we specified in the Target Machine'sboot.ini. COM1 is the defaultserial port, so we don't need to specify that.

Let's suppose that we've put all the symbols and source on the Sysphusserver. Here are the environment variables we need to set up on the HostMachine.

_NT_DEBUG_BAUD_RATE=115200
_NT_SYMBOL_PATH=SRV*\\Sysphus\softshared\symbols\SymStore*http://msdl.microsoft.com/download/symbols
_NT_ALT_SYMBOL_PATH=\\Sysphus\softshared\symbols\KillerApp\sym
_NT_SOURCE_PATH=\\Sysphus\softshared\symbols\KillerApp\sym

Figure 3 illustrates the Source and Symbol configuration described by theseenvironment settings. The debugger needs to find both the symbols and the sourcefor the binaries running on the Target. We don't particularly care how they gotonto the server. They could have been transferred to the server from a build onthe Host Machine, or they may have been archived here. If the debugger runningon the Host needs to match up OS symbols it will first look inSymstore on the Sysphus server.If this search fails, the debugger will attempt to transfer the symbols from theMicrosoft symbol server. Storing OS symbols on a local server means that otherdevelopers will not have to download symbols from the Microsoft server -- theycan point their debugger at the server and retrieve the local copies. This savesboth time and disk space.

Figure 3. Symbols and Source.

Let's Go!

Let's just test that this actually works. I have my latest drivers installedon the Target Machine and WinDbg is waiting to be invoked on the Host. Thehardware is waiting to be plugged in and everything's ready to roll. The targetis turned off.

• Invoke WinDbg on the Host. Enter this command from a command prompt:

C:\>WinDbg -k

WinDbg should start up and display

Microsoft (R) Windows Debugger Version6.1.0017.2
Copyright (c) Microsoft Corporation. All rightsreserved.
Opened \\.\com1
Waiting to reconnect...

• Turn On the Target. WinDbg will display the following

Connected to Windows XP 2600 x86 compatibletarget, ptr64 FALSE
Kernel Debugger connection established.
Symbol searchpath is:\\Sysphus\softshared\symbols\KillerApp\sym;SRV*\\Sysphus\softshared\symbols\SymStore*http://msdl.microsoft.com/download/symbols
Executablesearch path is:
Windows XP Kernel Version 2600 MP (1 procs) Checked x86compatible
Built by: 2600.xpsp1.020828-1920

• Set a break point

bp SaiNtHid!DriverEntry

• Insert the device

• We're in business.

Figure 4. Contact!

The debugger has correctly grabbedthe source from the server. Now let's ponder how we can leverage the server forsome real power debugging.

Adding a Remote Host

You may have wondered why I depicted the Host machine as a laptop. If youhave a building full of computers, any of which is likely to have some weirdinteraction with your driver, using a laptop is a quick and easy way of enablingremote debugging on any machine it's attached to. We need only make a slightchange to the scenario we have already examined. Our invocation of WinDbg isgoing to have to change to reflect a new role for the Host: we need to tellWinDbg that it has been promoted to a server.

To start the debug server we need a TCP/IP socket number. Your networkadministrator will be able to give you a valid number, but be prepared forlengthy discussions about firewalls and network security. Doughnuts should help.The socket number I have been given for this Host Machine is 3001. If you AREthe network administrator, or are entrusted with the grave responsibility ofselecting your own socket number, see the links in theAssigning a TCP/IP Socket Number Note.

NOTE - Assigning a TCP/IP Socket Number

These links should help you figure out which TCP/IP sockets to use:

http://www.iana.org/assignments/port-numbers

http://www.faqs.org/rfcs/rfc793.html

http://www.petri.co.il/tcp_ip_links_and_articles.htm

You can still control your debug session from the Host computer. The role ofthe Host has been enhanced, rather than subverted, by setting it up as a server.A second developer can now join the debug session, as illustrated in Figure5.

Figure 5. Adding a Remote Host.

The second developer can now join the debug session on the \\DEEPTHOUGHTRemote Host computer by issuing this command from a command prompt:

C:\>windbg -remotetcp:server=\\Jolwhqldev-xp,port=3001 -srcpath\\Sysphus\softshared\symbols\KillerApp\src

This syntax should be self explanatory, but it is worth noting that theRemote Host does not need to set up the symbols. The association between thebinaries on the Target and the Symbols on the Server is set up by the "real"Host machine (the laptop in all my pictures). In addition to describing theconnection to the Host with the server name andport number, we need to ensure that our session canaccess the source on the server. Depending on the debugging scenario we maychoose to limit the role of the Host machine, see the Attack of The Drones notefor more on this.

WinDbg should respond with something like the following

nt!RtlpBreakWithStatusInstruction:
80aaadcccc int 3
DEEPTHOUGHT\jwr (tcp 10.1.2.4:3389) connected at Tue Feb 252003

Now that the Remote Host is connected we can break into the session and setbreak points in exactly the same way we did in our initial experiments.

Attack of The Drones

If the Host is not manned we can set it up as a "drone" computer. The Hostdoes not need the source at all in this situation. If the Host is a roaminglaptop, it is probably a very good idea to prevent the machine from havingaccess to the source. In fact we could avoid the overhead of a GUI by usingKD.EXE as our Host debugger instead of WinDbg. The WinDbg Session running on theRemote Host will connect to this session in exactly the same way, regardless ofwhether KD or WinDbg is running on the host. We can invoke KD with exactly thesame environment variable settings that we used in our initial configuration ofthe Host. (Of course, it is not necessary to set the _NT_SOURCE_PATH variablesince KD won't be accessing the source.)

You can also append the -noio flag to ensure maximum inscrutability from KD.This will ensure that the debugger generates no output on the server machine. Ifyou are unused to the KD debugger, this is certainly the preferred operatingmode.

The flags do differ slightly between WinDbg and KD. Since KD is neveranything but a kernel debugger, we don't need to use the -k switch. However -kis used to specify the Target connection. The following examples shouldillustrate the point.

• To launch KD on the Host, using TCP port 3001 and COM1:

C:\>kd -server tcp:port=3001 -noio

• To launch KD on the Host, using port 3002 and COM2 instead (here we need the -k switch):

C:\>kd -server tcp:port=3002 -kcom:port=com2 -noio

I tend to have a machine set up like this with KD invoked in the registry runsection, so that a naive or occasional user can simply turn on the machine andstart a debug session, to which a developer can attach from a RemoteHost.

Adding Multiple Remote Hosts

You can extend the single Remote Host concept to allow Multiple Remote Hoststo join the debugging session. The scenario I presented above is geared towardthe lone developer. Additional developers can join the same debug session inexactly the same way. Figure 6 shows an additional Remote Host on thenetwork.

Figure 6. Adding Another Remote Host.

We can connect to the Real Host is exactly the same as we did in "Adding a Remote Host". You can also connect to thesession using the WinDbg GUI.

Using the WinDbg GUI to connect to a Remote Session

Using the GUI to connect to the Remote Host machine is only slightly morecomplex and arcane than using the command line. We need to set up the Server andSource Path, using two dialogs accessible in the file menu.

Step 1. Select Connect to a RemoteSession.

Adding Multiple Targets to a single Host

Adding a second Target machine is also pretty straightforward. All we need isa second com port and a second TCP/IP socket. Initially we used com1 to connectto the Target, so we will use com2 to connect up the Second Target machine. Wewill need to invoke a separate WinDbg for each Target. To check that this workswe can start up the Host for the COM1 Target using the steps inReview of the Standard Target and Host Setup.

Here are the additional steps to prepare for debugging of a secondTarget:

• First we need to set up the new Target for local debugging on the Host. This will enable us to verify that the serial connections between the Host and new Target are correct. Recall that WinDbg defaults to COM1. Our existing Target is using COM1. The new Target will use COM2. So the invocation for the second instance of WinDbg becomes:-

C:\>WinDbg -k com:port=Com2

Now we have two debug sessions running on the Host.

• To enable Remote Hosts to connect to either of these Debug Sessions We need a second TCP/IP socket. Recall that our original Target was assigned the socket 3001. I am going to associate 3002 with the second Target attached to com2.

  Shut down the new, local only, debug session and type the following:-

C:\>WinDbg -server tcp: port=3002-k com:port=com2

Now multiple Remote Hosts can connect to either of these debug sessions byspecifying the correct TCP/IP socket. This is an ideal scenario for a Test Lab.A single computer can host a debug session for each test machine, using multiplecom ports.

Figure 7 shows a basic Multiple Target setup.

Why would I want to do this?

Let's look at a concrete example of the multiple Target setup in action.Let's suppose I have some code that demonstrates an OS specific spin lock. (Infact, I'm using the SPINLOCK sample that accompaniesProgramming theMicrosoft Windows Driver Model Second Edition (Microsoft Press 2003).) Ifthe code is running on XP, it's going to use the more efficient in-stack queuedspin lock, otherwise it's going to degrade gracefully and use the old-style spinlock. I am going to prove that this works by stepping through the codesimultaneously on windows 2000 and XP.

The Target Machine on the left in Figure 7 is running Windows XP. The machineon the right has Windows 2000 installed. I'll use ports 3001 and 3002 to handlethe remote debugging sessions for these machines, respectively.

Let's assume that the debuggers on the Host have been set up and areconnected up correctly to com1 and com2. I am going to focus on how we get twodebug sessions running on a Remote Host. Here wego....    

• Install the driver on both Target machines. In this example, you use the Add Hardware wizard for a fake device whose driver has an INF file.
• Turn off both Target machines.

• On the Remote Host, run an instance of the debugger for each Target we want to debug.

  Port 3001, the XP Target machine on com1:

C:\>windbg -remotetcp:server=\\Jolwhqldev-xp,port=3001 -srcpath\\Sysphus\softshared\symbols\KillerApp\src

Port 3002, the Windows 2000 Target machine on com1:

C:\>windbg -remotetcp:server=\\Jolwhqldev-xp,port=3002 -srcpath\\Sysphus\softshared\symbols\KillerApp\src

• Start both Targets.

• During the boot sequence break into each Remote Host debugger instance and set a breakpoint:

bu spinlock!InitializeSpinLockFunctionPointers

• Restart the Debuggers.

g

• Both debuggers will break when execution reaches the breakpoint we set. We can single step through the code and prove irrefutably that this code will indeed only try to use queued spin locks on Windows XP. figure 8 is a screen shot showing two instances of WinDbg running on my Remote Host, with execution paused at the OS-specific code.

The debugger correctly illustrates the different code paths on the differentoperating systems. 

 

Summary

In this article I have a tried to steer a straight course from the simplestconfiguration to a point where we are ready to embark on some profoundly complexand esoteric configurations. We could employ these techniques, for example, torig up a whole Test Lab to a single Host Computer. An army of developers couldbe ready and waiting to attach remotely to any Target Machine when a problemarises. The days of wandering around with a laptop and a Null Modem cable are welland truly over.

As well as enabling development/ test teams to work more efficiently,using WinDbg in this way is a fantastic learning tool. As I demonstrated in thelast example, simultaneoulsy stepping through the same code on differentoperating systems is now a possibility. When you consider that any otherdeveloper with access to the magic socket could also join the debug session, thepossibilities are endless. Whether you are a single developer, whose code needsto run on a plethora of different software/ hardware configurations, or part ofa vast team of developers, these techniques can be invaluable.

About the author:

Jolyon Wright is Principal Software Engineer atSaitek plc, purveyors of finehardware and software solutions for the gaming industry since 1979.