freebsd-dev/share/doc/handbook/kerneldebug.sgml
Doug Rabson 9e5f720f5f Document how to debug LKMs with kgdb.
Reviewed by:	J Wunsch <j@uriah.heep.sax.de>
1997-04-06 23:03:27 +00:00

526 lines
21 KiB
Plaintext

<!-- $Id: kerneldebug.sgml,v 1.13 1997/03/18 00:42:36 joerg Exp $ -->
<!-- The FreeBSD Documentation Project -->
<chapt><heading>Kernel Debugging<label id="kerneldebug"></heading>
<p><em>Contributed by &a.paul; and &a.joerg;</em>
<sect><heading>Debugging a kernel crash dump with kgdb</heading>
<p>Here are some instructions for getting kernel debugging
working on a crash dump, it assumes that you have enough swap
space for a crash dump. If you have multiple swap
partitions and the first one is too small to hold the dump,
you can configure your kernel to use an alternate dump device
(in the <tt>config kernel</tt> line), or
you can specify an alternate using the dumpon(8) command.
Dumps to non-swap devices,
tapes for example, are currently not supported. Config your
kernel using <tt>config -g</tt>.
See <ref id="kernelconfig" name="Kernel Configuration"> for
details on configuring the FreeBSD kernel.
Use the <tt>dumpon(8)</tt> command to tell the kernel where to dump
to (note that this will have to be done after configuring the
partition in question as swap space via <tt>swapon(8)</tt>). This is
normally arranged via <tt>/etc/sysconfig</tt> and <tt>/etc/rc</tt>.
Alternatively, you can
hard-code the dump device via the `dump' clause in the `config' line
of your kernel config file. This is deprecated, use only if you
want a crash dump from a kernel that crashes during booting.
<em><bf>Note:</bf> In the following, the term `<tt>kgdb</tt>' refers
to <tt>gdb</tt> run in `kernel debug mode'. This can be accomplished by
either starting the <tt>gdb</tt> with the option <tt>-k</tt>, or by linking
and starting it under the name <tt>kgdb</tt>. This is not being
done by default, however, and the idea is basically deprecated since
the GNU folks do not love it if their tools behave differently when
called by another name. This feature might as well be discontinued
in further releases.</em>
When the kernel has been built make a copy of it, say
<tt>kernel.debug</tt>, and then run <tt>strip -d</tt> on the
original. Install the original as normal. You may also install
the unstripped kernel, but symbol table lookup time for some
programs will drastically increase, and since
the whole kernel is loaded entirely at boot time and cannot be
swapped out later, several megabytes of
physical memory will be wasted.
If you are testing a new kernel, for example by typing the new
kernel's name at the boot prompt, but need to boot a different
one in order to get your system up and running again, boot it
only into single user state using the <tt>-s</tt> flag at the
boot prompt, and then perform the following steps:
<tscreen><verb>
fsck -p
mount -a -t ufs # so your file system for /var/crash is writable
savecore -N /kernel.panicked /var/crash
exit # ...to multi-user
</verb></tscreen>
This instructs <tt>savecore(8)</tt> to use another kernel for symbol name
extraction. It would otherwise default to the currently running kernel
and most likely not do anything at all since the crash dump and the
kernel symbols differ.
Now, after a crash dump, go to <tt>/sys/compile/WHATEVER</tt> and run
<tt>kgdb</tt>. From <tt>kgdb</tt> do:
<tscreen><verb>
symbol-file kernel.debug
exec-file /var/crash/kernel.0
core-file /var/crash/vmcore.0
</verb></tscreen>
and voila, you can debug the crash dump using the kernel sources
just like you can for any other program.
Here is a script log of a <tt>kgdb</tt> session illustrating the
procedure. Long
lines have been folded to improve readability, and the lines are
numbered for reference. Despite this, it is a real-world error
trace taken during the development of the pcvt console driver.
<tscreen><verb>
1:Script started on Fri Dec 30 23:15:22 1994
2:uriah # cd /sys/compile/URIAH
3:uriah # kgdb kernel /var/crash/vmcore.1
4:Reading symbol data from /usr/src/sys/compile/URIAH/kernel...done.
5:IdlePTD 1f3000
6:panic: because you said to!
7:current pcb at 1e3f70
8:Reading in symbols for ../../i386/i386/machdep.c...done.
9:(kgdb) where
10:#0 boot (arghowto=256) (../../i386/i386/machdep.c line 767)
11:#1 0xf0115159 in panic ()
12:#2 0xf01955bd in diediedie () (../../i386/i386/machdep.c line 698)
13:#3 0xf010185e in db_fncall ()
14:#4 0xf0101586 in db_command (-266509132, -266509516, -267381073)
15:#5 0xf0101711 in db_command_loop ()
16:#6 0xf01040a0 in db_trap ()
17:#7 0xf0192976 in kdb_trap (12, 0, -272630436, -266743723)
18:#8 0xf019d2eb in trap_fatal (...)
19:#9 0xf019ce60 in trap_pfault (...)
20:#10 0xf019cb2f in trap (...)
21:#11 0xf01932a1 in exception:calltrap ()
22:#12 0xf0191503 in cnopen (...)
23:#13 0xf0132c34 in spec_open ()
24:#14 0xf012d014 in vn_open ()
25:#15 0xf012a183 in open ()
26:#16 0xf019d4eb in syscall (...)
27:(kgdb) up 10
28:Reading in symbols for ../../i386/i386/trap.c...done.
29:#10 0xf019cb2f in trap (frame={tf_es = -260440048, tf_ds = 16, tf_\
30:edi = 3072, tf_esi = -266445372, tf_ebp = -272630356, tf_isp = -27\
31:2630396, tf_ebx = -266427884, tf_edx = 12, tf_ecx = -266427884, tf\
32:_eax = 64772224, tf_trapno = 12, tf_err = -272695296, tf_eip = -26\
33:6672343, tf_cs = -266469368, tf_eflags = 66066, tf_esp = 3072, tf_\
34:ss = -266427884}) (../../i386/i386/trap.c line 283)
35:283 (void) trap_pfault(&amp;frame, FALSE);
36:(kgdb) frame frame->tf_ebp frame->tf_eip
37:Reading in symbols for ../../i386/isa/pcvt/pcvt_drv.c...done.
38:#0 0xf01ae729 in pcopen (dev=3072, flag=3, mode=8192, p=(struct p\
39:roc *) 0xf07c0c00) (../../i386/isa/pcvt/pcvt_drv.c line 403)
40:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
41:(kgdb) list
42:398
43:399 tp->t_state |= TS_CARR_ON;
44:400 tp->t_cflag |= CLOCAL; /* cannot be a modem (:-) */
45:401
46:402 #if PCVT_NETBSD || (PCVT_FREEBSD >= 200)
47:403 return ((*linesw[tp->t_line].l_open)(dev, tp));
48:404 #else
49:405 return ((*linesw[tp->t_line].l_open)(dev, tp, flag));
50:406 #endif /* PCVT_NETBSD || (PCVT_FREEBSD >= 200) */
51:407 }
52:(kgdb) print tp
53:Reading in symbols for ../../i386/i386/cons.c...done.
54:$1 = (struct tty *) 0x1bae
55:(kgdb) print tp->t_line
56:$2 = 1767990816
57:(kgdb) up
58:#1 0xf0191503 in cnopen (dev=0x00000000, flag=3, mode=8192, p=(st\
59:ruct proc *) 0xf07c0c00) (../../i386/i386/cons.c line 126)
60: return ((*cdevsw[major(dev)].d_open)(dev, flag, mode, p));
61:(kgdb) up
62:#2 0xf0132c34 in spec_open ()
63:(kgdb) up
64:#3 0xf012d014 in vn_open ()
65:(kgdb) up
66:#4 0xf012a183 in open ()
67:(kgdb) up
68:#5 0xf019d4eb in syscall (frame={tf_es = 39, tf_ds = 39, tf_edi =\
69: 2158592, tf_esi = 0, tf_ebp = -272638436, tf_isp = -272629788, tf\
70:_ebx = 7086, tf_edx = 1, tf_ecx = 0, tf_eax = 5, tf_trapno = 582, \
71:tf_err = 582, tf_eip = 75749, tf_cs = 31, tf_eflags = 582, tf_esp \
72:= -272638456, tf_ss = 39}) (../../i386/i386/trap.c line 673)
73:673 error = (*callp->sy_call)(p, args, rval);
74:(kgdb) up
75:Initial frame selected; you cannot go up.
76:(kgdb) quit
77:uriah # exit
78:exit
79:
80:Script done on Fri Dec 30 23:18:04 1994
</verb></tscreen>
Comments to the above script:
<descrip>
<tag/line 6:/ This is a dump taken from within DDB (see below), hence the
panic comment ``because you said to!'', and a rather long
stack trace; the initial reason for going into DDB has been
a page fault trap though.
<tag/line 20:/ This is the location of function <tt>trap()</tt>
in the stack trace.
<tag/line 36:/ Force usage of a new stack frame; this is no longer
necessary now. The stack frames are supposed to point to
the right locations now, even in case of a trap.
(I do not have a new core dump handy &lt;g&gt;, my kernel
did not panic for ia rather long time.)
From looking at the code in source line 403,
there is a high probability that either the pointer
access for ``tp'' was messed up, or the array access was
out of bounds.
<tag/line 52:/ The pointer looks suspicious, but happens to be a valid
address.
<tag/line 56:/ However, it obviously points to garbage, so we have found our
error! (For those unfamiliar with that particular piece
of code: <tt>tp-&gt;t_line</tt> refers to the line discipline
of the console device here, which must be a rather small integer
number.)
</descrip>
<sect><heading>Post-mortem analysis of a dump</heading>
<p>What do you do if a kernel dumped core but you did not expect
it, and it is therefore not compiled using <tt>config -g</tt>?
Not everything is lost here. Do not panic!
Of course, you still need to enable crash dumps. See above
on the options you have to specify in order to do this.
Go to your kernel compile directory, and edit the line
containing <tt>COPTFLAGS?=-O</tt>. Add the <tt>-g</tt> option
there (but <em>do not</em> change anything on the level of
optimization). If you do already know roughly the probable
location of the failing piece of code (e.g., the <tt>pcvt</tt>
driver in the example above), remove all the object files for
this code. Rebuild the kernel. Due to the time stamp change on
the Makefile, there will be some other object files rebuild,
for example <tt>trap.o</tt>. With a bit of luck, the added
<tt>-g</tt> option will not change anything for the generated
code, so you will finally get a new kernel with similar code to
the faulting one but some debugging symbols. You should at
least verify the old and new sizes with the <tt>size(1)</tt> command. If
there is a mismatch, you probably need to give up here.
Go and examine the dump as described above. The debugging
symbols might be incomplete for some places, as can be seen in
the stack trace in the example above where some functions are
displayed without line numbers and argument lists. If you need
more debugging symbols, remove the appropriate object files and
repeat the <tt>kgdb</tt> session until you know enough.
All this is not guaranteed to work, but it will do it fine in
most cases.
<sect><heading>On-line kernel debugging using DDB</heading>
<p>While <tt>kgdb</tt> as an offline debugger provides a very
high level of user interface, there are some things it cannot do.
The most important ones being breakpointing and single-stepping
kernel code.
If you need to do low-level debugging on your kernel, there is
an on-line debugger available called DDB. It allows to
setting breakpoints, single-steping kernel functions, examining
and changing kernel variables, etc. However, it cannot not
access kernel source files, and only has access to the global
and static symbols, not to the full debug information like
<tt>kgdb</tt>.
To configure your kernel to include DDB, add the option line
<tscreen><verb>
options DDB
</verb></tscreen>
to your config file, and rebuild. (See <ref id="kernelconfig"
name="Kernel Configuration"> for details on configuring the
FreeBSD kernel. Note that if you have an older version of the
boot blocks, your debugger symbols might not be loaded at all.
Update the boot blocks, the recent ones do load the DDB symbols
automagically.)
Once your DDB kernel is running, there are several ways to
enter DDB. The first, and earliest way is to type the boot
flag <tt>-d</tt> right at the boot prompt. The kernel will
start up in debug mode and enter DDB prior to any device
probing. Hence you are able to even debug the device
probe/attach functions.
The second scenario is a hot-key on the keyboard, usually
Ctrl-Alt-ESC. For syscons, this can be remapped, and some of
the distributed maps do this, so watch out.
There is an option
available for serial consoles
that allows the use of a serial line BREAK on the console line to
enter DDB (``<tt>options BREAK_TO_DEBUGGER</tt>''
in the kernel config file). It is not the default since there are a lot of
crappy serial adapters around that gratuitously generate a
BREAK condition for example when pulling the cable.
The third way is that any panic condition will branch to DDB if
the kernel is configured to use it.
For this reason, it is not wise to
configure a kernel with DDB for a machine running unattended.
The DDB commands roughly resemble some <tt>gdb</tt> commands. The first you
probably need is to set a breakpoint:
<tscreen><verb>
b function-name
b address
</verb></tscreen>
Numbers are taken hexadecimal by default, but to make them
distinct from symbol names, hexadecimal numbers starting with the
letters <tt>a</tt>-<tt>f</tt> need to be preceded with
<tt>0x</tt> (for other numbers, this is optional). Simple
expressions are allowed, for example: <tt>function-name + 0x103</tt>.
To continue the operation of an interrupted kernel, simply type
<tscreen><verb>
c
</verb></tscreen>
To get a stack trace, use
<tscreen><verb>
trace
</verb></tscreen>
Note that when entering DDB via a hot-key, the kernel is currently
servicing an interrupt, so the stack trace might be not of much use
for you.
If you want to remove a breakpoint, use
<tscreen><verb>
del
del address-expression
</verb></tscreen>
The first form will be accepted immediately after a breakpoint hit,
and deletes the current breakpoint. The second form can remove any
breakpoint, but you need to specify the exact address, as it can be
obtained from
<tscreen><verb>
show b
</verb></tscreen>
To single-step the kernel, try
<tscreen><verb>
s
</verb></tscreen>
This will step into functions, but you can make DDB trace them until
the matching return statement is reached by
<tscreen><verb>
n
</verb></tscreen>
<bf>Note:</bf> this is different from <tt>gdb</tt>'s `next' statement, it is like
<tt>gdb</tt>'s `finish'.
To examine data from memory, use (for example):
<tscreen><verb>
x/wx 0xf0133fe0,40
x/hd db_symtab_space
x/bc termbuf,10
x/s stringbuf
</verb></tscreen>
for word/halfword/byte access, and hexadecimal/decimal/character/
string display. The number after the comma is the object count.
To display the next 0x10 items, simply use
<tscreen><verb>
x ,10
</verb></tscreen>
Similarly, use
<tscreen><verb>
x/ia foofunc,10
</verb></tscreen>
to disassemble the first 0x10 instructions of <tt>foofunc</tt>, and display
them along with their offset from the beginning of <tt>foofunc</tt>.
To modify the memory, use the write command:
<tscreen><verb>
w/b termbuf 0xa 0xb 0
w/w 0xf0010030 0 0
</verb></tscreen>
The command modifier (<tt>b</tt>/<tt>h</tt>/<tt>w</tt>)
specifies the size of the data to be written, the first
following expression is the address to write to, the remainder
is interpreted as data to write to successive memory locations.
If you need to know the current registers, use
<tscreen><verb>
show reg
</verb></tscreen>
Alternatively, you can display a single register value by e.g.
<tscreen><verb>
p $eax
</verb></tscreen>
and modify it by
<tscreen><verb>
set $eax new-value
</verb></tscreen>
Should you need to call some kernel functions from DDB, simply
say
<tscreen><verb>
call func(arg1, arg2, ...)
</verb></tscreen>
The return value will be printed.
For a <tt>ps(1)</tt> style summary of all running processes, use
<tscreen><verb>
ps
</verb></tscreen>
Now you have now examined why your kernel failed, and you wish to
reboot. Remember that, depending on the severity of previous
malfunctioning, not all parts of the kernel might still be working
as expected. Perform one of the following actions to shut down and
reboot your system:
<tscreen><verb>
call diediedie()
</verb></tscreen>
will cause your kernel to dump core and reboot, so you can
later analyze the core on a higher level with kgdb. This
command usually must be followed by another
`<tt>continue</tt>' statement.
There is now an alias for this: `<tt>panic</tt>'.
<tscreen><verb>
call boot(0)
</verb></tscreen>
might be a good way to cleanly shut down the running system, <tt>sync()</tt>
all disks, and finally reboot. As long as the disk and file system
interfaces of the kernel are not damaged, this might be a good way
for an almost clean shutdown.
<tscreen><verb>
call cpu_reset()
</verb></tscreen>
is the final way out of disaster and almost the same as hitting
the Big Red Button.
If you need a short command summary, simply type
<tscreen><verb>
help
</verb></tscreen>
However, it is highly recommended to have a printed copy of the
<tt>ddb(4)</tt> manual page ready for a debugging session.
Remember that it is hard to read the on-line manual while
single-stepping the kernel.
<sect><heading>On-line kernel debugging using remote GDB</heading>
<p>This feature is supported since FreeBSD 2.2, and it's actually
a very neat one.
GDB used to support <em/remote debugging/ for a long time
already. This is done using a very simple protocol along a
serial line. Obviously, and opposed to the other methods
described above, you need two machines for doing this. One is
the host providing the debugging environment, including all
the sources, and a copy of the kernel binary with all the
symbols in it, and the other one is the target machine that
simply runs a similar copy of the very same kernel (but stripped
off the debugging information).
You should configure the kernel in question with <tt>config -g</tt>,
include <em/DDB/ into the configuration, and compile it as usual.
This gives a large blurb of a binary, due
to the debugging information. Copy this kernel to the target
machine, strip the debugging symbols off with <tt>strip -x</tt>,
and boot it using the <tt/-d/ boot option. Connect the first
serial line of the target machine to any serial line of the
debugging host. Now, on the debugging machine, go to the compile
directory of the target kernel, and start gdb:
<tscreen><verb>
% gdb -k kernel
GDB is free software and you are welcome to distribute copies of it
under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details.
GDB 4.16 (i386-unknown-freebsd),
Copyright 1996 Free Software Foundation, Inc...
(kgdb)
</verb></tscreen>
Initialize the remote debugging session (assuming the first serial
port is being used) by:
<tscreen><verb>
(kgdb) target remote /dev/cuaa0
</verb></tscreen>
Now, on the target host (that entered DDB right before even starting
the device probe), type:
<tscreen><verb>
Debugger("Boot flags requested debugger")
Stopped at Debugger+0x35: movb $0, edata+0x51bc
db> gdb
</verb></tscreen>
DDB will respond with:
<tscreen><verb>
Next trap will enter GDB remote protocol mode
</verb></tscreen>
Every time you type ``gdb'', the mode will be toggled between
remote GDB and local DDB. In order to force a next trap
immediately, simply type ``s'' (step). Your hosting GDB will
now gain control over the target kernel:
<tscreen><verb>
Remote debugging using /dev/cuaa0
Debugger (msg=0xf01b0383 "Boot flags requested debugger")
at ../../i386/i386/db_interface.c:257
(kgdb)
</verb></tscreen>
You can use this session almost as any other GDB session, including
full access to the source, running it in gud-mode inside an Emacs
window (which gives you an automatic source code display in another
Emacs window) etc.
<p>Remote GDB can also be used to debug LKMs. First build the LKM
with debugging symbols:
<tscreen><verb>
# cd /usr/src/lkm/linux
# make clean; make COPTS=-g
</verb></tscreen>
Then install this version of the module on the target machine, load it
and use <tt>modstat</tt> to find out where it was loaded:
<tscreen><verb>
# linux
# modstat
Type Id Off Loadaddr Size Info Rev Module Name
EXEC 0 4 f5109000 001c f510f010 1 linux_mod
</verb></tscreen>
Take the load address of the module and add 0x20 (probably to account
for the a.out header). This is the address that the module code was
relocated to. Use the <tt>add-symbol-file</tt> command in GDB to tell the
debugger about the module:
<tscreen><verb>
(kgdb) add-symbol-file /usr/src/lkm/linux/linux_mod.o 0xf5109020
add symbol table from file "/usr/src/lkm/linux/linux_mod.o" at
text_addr = 0xf5109020?
(y or n) y
(kgdb)
</verb></tscreen>
You now have access to all the symbols in the LKM.
<sect><heading>Debugging a console driver</heading>
<p>Since you need a console driver to run DDB on, things are more
complicated if the console driver itself is failing. You might
remember the use of a serial console (either with modified boot
blocks, or by specifying <tt><bf>-h</bf></tt> at the <tt>Boot:</tt>
prompt), and hook up a standard
terminal onto your first serial port. DDB works on any configured
console driver, of course also on a serial console.