Windows Server 2003 (NT 5.2.3790.0) build guide
Debugging
Kernel debugging can be enabled by editing the C:\boot.ini file in your installed build, and adding /debug /debugport=COM1 /baudrate=115200 to the end of a config line.
For example here’s a boot.ini with two choices, one with debugging & one without, with this NTLDR will show an OS boot choice menu at startup:
[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /debug /debugport=COM1 /baudrate=115200
(even though they’re named the same Windows Server 2003, Enterprise, NTLDR will add a [debugger enabled] tag to it on the OS selection menu.)
chk (checked) builds are preferred for debugging as they enable asserts & debug messages, though chk will run a lot slower than fre builds. fre builds can also be debugged fine, but usually with a lot less debug output.
Ideally you should also configure WinDbg to use the binaries.{buildtype}\symbols.pri\retail & binaries.{buildtype}\symbols\retail symbols folders too, if WinDbg is running on the same machine that compiled the build it should then also be able to do source-level debugging against the actual source files.
Note that if you’re using a VM to host your build on you’ll have to pass-through the emulated COM port over to a pipe somehow, for WinDbg/KD/IDA/etc to make use of.
There is apparently a way to get KDNET working on 2003 too (see https://github.com/MovAX0xDEAD/KDNET), but this is as-of-yet untested with our builds, seems like it should have support for VMware’s emulated network hardware at least.
(TODO: add more VM guides here… if anyone wants to post one in the thread I’m happy to add it here)
VMware setup
For a VMware VM, just open the VM hardware options, remove the printer hardware, then click Add... and choose Serial Port.
In the serial-port config panel, set it to Use named pipe, and enter a pipe name into the textbox, eg. \\.\pipe\vm. The pipe should be setup as This end is the server & The other end is a virtual machine.
It’s apparently recommended to use the Yield CPU on poll option, but I’ve had success without it, it’s up to you.
With that setup now just open WinDbg, choose Attach to kernel, open the COM tab and enter the baud-rate and pipe name you selected for your VM, then hit OK.
Now WinDbg should start with Waiting for reconnect text showing, and hopefully when you next boot up your VM it’ll connect up to WinDbg by itself.
Debugging Setup/Installation
A debugger can be attached to setup by pressing F8 at the right time - for text-mode setup the right time is when asked Press F6 if you need to install a third party SCSI or RAID driver, spamming it while that message is showing should make it connect to COM1 with 19200 baud-rate (debug options can be changed inside txtsetup.sif, the SetupDebugOptions line specifies options to apply when F8 is pressed). After debug connection is made it might take a couple extra minutes for it to pass the Setup is starting Windows stage.
Similarly, GUI setup can be debugged by pressing F8 before the Windows bootloader starts (progress bar etc), this should bring up a menu with options for Safe Mode, Debugging Mode, etc. Choosing Debugging Mode will make it attach to COM1 before starting setup, again at 19200 baudrate.
Changing default baudrate
The default 19200 baudrate can be changed by editing the base\boot\kdcom\xxkdsup.c file, search for BD_19200 inside there and change it to e.g. BD_115200, now it should use that by default without needing any /baudrate parameter, or eg. when using Debugging Mode from the boot options menu.
Un-filtering kernel messages
Even with a chk build you may notice that the kernel doesn’t seem to output much over KD, this seems to be because most of the kernel components use KdPrintEx, which allows filtering KD messages to certain components (although MS’s docs seem to suggest KdPrintEx filtering was only used in Vista, it does seem that 2003 makes use of it too)
To enable components you’ll need to open the registry of your installed build to the following key (or create it if it doesn’t exist)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Debug Print Filter\
Then create a DWORD value named the component you want to enable (eg. LDR), and set the value to hexadecimal FFFFFFFF.
A list of the component registry names can be found inside base\published\obj\i386\dpfiltercm.c after a build. This also contains the symbol names for the component (eg. Kd_LDR_Mask), which can be set directly through WinDbg/KD via the symbol name. (eg. ed nt!Kd_LDR_Mask 0xFFFFFFFF, with symbols loaded this should set the LDR components filter value in real-time)
The WIN2000 components value (default 1) is added to every components value, essentially making WIN2000 the default value for all components - so setting this to FFFFFFFF should make every component print to KD. You’ll get a ton more debug output from this, but all the KD prints will likely slow down the system a lot (unfortunately it doesn’t seem possible to disable a component when using WIN2000 to enable them all)
If you need to enable components before your build is installed (eg. kernel is having problems before setup is completed), you’ll need to edit the mergedcomponents\setupinfs\hivesys.inx file, anon made a post about that here, though it hasn’t been tried yet afaik.