Add a script to perform simple analysis of a crash dump (either a full
dump or minidump). When the script is run, it generates a text file containing the output of several commands run againt the core dump such as kgdb (stack trace), ps, netstat, vmstat, iostat, dmesg, and fstat. Obtained from: Yahoo! MFC after: 2 weeks
This commit is contained in:
John Baldwin
parent
da7bbd2c08
commit
a37f97161c
@ -33,6 +33,7 @@ SUBDIR= ac \
|
||||
ckdist \
|
||||
clear_locks \
|
||||
config \
|
||||
crashinfo \
|
||||
cron \
|
||||
crunch \
|
||||
ctm \
|
||||
|
||||
6
usr.sbin/crashinfo/Makefile
Normal file
6
usr.sbin/crashinfo/Makefile
Normal file
@ -0,0 +1,6 @@
|
||||
# $FreeBSD$
|
||||
|
||||
SCRIPTS= crashinfo.sh
|
||||
MAN= crashinfo.8
|
||||
|
||||
.include <bsd.prog.mk>
|
||||
109
usr.sbin/crashinfo/crashinfo.8
Normal file
109
usr.sbin/crashinfo/crashinfo.8
Normal file
@ -0,0 +1,109 @@
|
||||
.\" Copyright (c) 2008 Yahoo!, Inc.
|
||||
.\" All rights reserved.
|
||||
.\"
|
||||
.\" Redistribution and use in source and binary forms, with or without
|
||||
.\" modification, are permitted provided that the following conditions
|
||||
.\" are met:
|
||||
.\" 1. Redistributions of source code must retain the above copyright
|
||||
.\" notice, this list of conditions and the following disclaimer.
|
||||
.\" 2. Redistributions in binary form must reproduce the above copyright
|
||||
.\" notice, this list of conditions and the following disclaimer in the
|
||||
.\" documentation and/or other materials provided with the distribution.
|
||||
.\" 3. Neither the name of the author nor the names of any co-contributors
|
||||
.\" may be used to endorse or promote products derived from this software
|
||||
.\" without specific prior written permission.
|
||||
.\"
|
||||
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
||||
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
||||
.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
||||
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
||||
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
||||
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
||||
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
||||
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
||||
.\" SUCH DAMAGE.
|
||||
.\"
|
||||
.\" $FreeBSD$
|
||||
.\"
|
||||
.Dd June 28, 2008
|
||||
.Dt CRASHINFO 8
|
||||
.Os
|
||||
.Sh NAME
|
||||
.Nm crashinfo
|
||||
.Nd "analyze a core dump of the operating system"
|
||||
.Sh SYNOPSIS
|
||||
.Nm
|
||||
.Op Fl d Ar crashdir
|
||||
.Op Fl n Ar dumpnr
|
||||
.Op Fl k Ar kernel
|
||||
.Op Ar core
|
||||
.Sh DESCRIPTION
|
||||
The
|
||||
.Nm
|
||||
utility analyzes a core dump saved by
|
||||
.Xr savecore 8 .
|
||||
It generates a text file containing the analysis in the same directory as
|
||||
the core dump.
|
||||
For a given core dump file named
|
||||
.Pa vmcore.XX
|
||||
the generated text file will be named
|
||||
.Pa core.txt.XX .
|
||||
.Pp
|
||||
By default,
|
||||
.Nm
|
||||
analyzes the most recent core dump in the core dump directory.
|
||||
A specific core dump may be specified via either the
|
||||
.Ar core
|
||||
or
|
||||
.Ar dumpnr
|
||||
arguments.
|
||||
Once
|
||||
.Nm
|
||||
has located a core dump,
|
||||
it analyzes the core dump to determine the exact version of the kernel
|
||||
that generated the core.
|
||||
It then looks for a matching kernel file under each of the subdirectories in
|
||||
.Pa /boot .
|
||||
The location of the kernel file can also be explicitly provided via the
|
||||
.Ar kernel
|
||||
argument.
|
||||
.Pp
|
||||
Once
|
||||
.Nm
|
||||
has located a core dump and kernel,
|
||||
it uses several utilities to analyze the core including
|
||||
.Xr dmesg 8 ,
|
||||
.Xr fstat 1 ,
|
||||
.Xr iostat 8 ,
|
||||
.Xr ipcs 1 ,
|
||||
.Xr kgdb 1 ,
|
||||
.Xr netstat 1 ,
|
||||
.Xr nfsstat 1 ,
|
||||
.Xr ps 1 ,
|
||||
.Xr pstat 8 ,
|
||||
and
|
||||
.Xr vmstat 8 .
|
||||
.Pp
|
||||
The options are as follows:
|
||||
.Bl -tag -width indent
|
||||
.It Fl d Ar crashdir
|
||||
Specify an alternate core dump directory.
|
||||
The default crash dump directory is
|
||||
.Pa /var/crash .
|
||||
.It Fl n Ar dumpnr
|
||||
Use the core dump saved in
|
||||
.Pa vmcore. Ns Ar dumpnr
|
||||
instead of the latest core in the core dump directory.
|
||||
.It Fl k Ar kernel
|
||||
Specify an explicit kernel file.
|
||||
.El
|
||||
.Sh SEE ALSO
|
||||
.Xr savecore 8 ,
|
||||
.Xr textdump 4
|
||||
.Sh HISTORY
|
||||
The
|
||||
.Nm
|
||||
utility appeared in
|
||||
.Fx 8.0 .
|
||||
306
usr.sbin/crashinfo/crashinfo.sh
Executable file
306
usr.sbin/crashinfo/crashinfo.sh
Executable file
@ -0,0 +1,306 @@
|
||||
#!/bin/sh
|
||||
#
|
||||
# Copyright (c) 2008 Yahoo!, Inc.
|
||||
# All rights reserved.
|
||||
#
|
||||
# Redistribution and use in source and binary forms, with or without
|
||||
# modification, are permitted provided that the following conditions
|
||||
# are met:
|
||||
# 1. Redistributions of source code must retain the above copyright
|
||||
# notice, this list of conditions and the following disclaimer.
|
||||
# 2. Redistributions in binary form must reproduce the above copyright
|
||||
# notice, this list of conditions and the following disclaimer in the
|
||||
# documentation and/or other materials provided with the distribution.
|
||||
# 3. Neither the name of the author nor the names of any co-contributors
|
||||
# may be used to endorse or promote products derived from this software
|
||||
# without specific prior written permission.
|
||||
#
|
||||
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
||||
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
||||
# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
||||
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
||||
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
||||
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
||||
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
||||
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
||||
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
||||
# SUCH DAMAGE.
|
||||
#
|
||||
# $FreeBSD$
|
||||
|
||||
usage()
|
||||
{
|
||||
echo "usage: crashinfo [-d crashdir] [-n dumpnr] [-k kernel] [core]"
|
||||
exit 1
|
||||
}
|
||||
|
||||
find_kernel()
|
||||
{
|
||||
local ivers k kvers
|
||||
|
||||
ivers=$(awk '
|
||||
/Version String/ {
|
||||
print
|
||||
nextline=1
|
||||
next
|
||||
}
|
||||
// {
|
||||
if (nextline) {
|
||||
print
|
||||
nextline=0
|
||||
}
|
||||
}' $INFO)
|
||||
|
||||
# Look for a matching kernel version.
|
||||
for k in /boot/kernel/kernel $(ls -t /boot/*/kernel); do
|
||||
kvers=$(echo 'printf " Version String: %s", version' | \
|
||||
gdb -x /dev/stdin -batch $k 2>/dev/null)
|
||||
if [ "$ivers" = "$kvers" ]; then
|
||||
KERNEL=$k
|
||||
break
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
CRASHDIR=/var/crash
|
||||
DUMPNR=
|
||||
KERNEL=
|
||||
|
||||
while getopts "d:n:k:" opt; do
|
||||
case "$opt" in
|
||||
d)
|
||||
CRASHDIR=$OPTARG
|
||||
;;
|
||||
n)
|
||||
DUMPNR=$OPTARG
|
||||
;;
|
||||
k)
|
||||
KERNEL=$OPTARG
|
||||
;;
|
||||
\?)
|
||||
usage
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
shift $((OPTIND - 1))
|
||||
|
||||
if [ $# -eq 1 ]; then
|
||||
if [ -n "$DUMPNR" ]; then
|
||||
echo "-n and an explicit vmcore are mutually exclusive"
|
||||
usage
|
||||
fi
|
||||
|
||||
# Figure out the crash directory and number from the vmcore name.
|
||||
CRASHDIR=`dirname $1`
|
||||
DUMPNR=$(expr $(basename $1) : 'vmcore\.\([0-9]*\)$')
|
||||
if [ -z "$DUMPNR" ]; then
|
||||
echo "Unable to determine dump number from vmcore file $1."
|
||||
exit 1
|
||||
fi
|
||||
elif [ $# -gt 1 ]; then
|
||||
usage
|
||||
else
|
||||
# If we don't have an explicit dump number, operate on the most
|
||||
# recent dump.
|
||||
if [ -z "$DUMPNR" ]; then
|
||||
if ! [ -r $CRASHDIR/bounds ]; then
|
||||
echo "No crash dumps in $CRASHDIR."
|
||||
exit 1
|
||||
fi
|
||||
next=`cat $CRASHDIR/bounds`
|
||||
if [ -z "$next" ] || [ "$next" -eq 0 ]; then
|
||||
echo "No crash dumps in $CRASHDIR."
|
||||
exit 1
|
||||
fi
|
||||
DUMPNR=$(($next - 1))
|
||||
fi
|
||||
fi
|
||||
|
||||
VMCORE=$CRASHDIR/vmcore.$DUMPNR
|
||||
INFO=$CRASHDIR/info.$DUMPNR
|
||||
FILE=$CRASHDIR/core.txt.$DUMPNR
|
||||
HOSTNAME=`hostname`
|
||||
|
||||
if [ ! -e $VMCORE ]; then
|
||||
echo "$VMCORE not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -e $INFO ]; then
|
||||
echo "$INFO not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# If the user didn't specify a kernel, then try to find one.
|
||||
if [ -z "$KERNEL" ]; then
|
||||
find_kernel
|
||||
if [ -z "$KERNEL" ]; then
|
||||
echo "Unable to find matching kernel for $VMCORE"
|
||||
exit 1
|
||||
fi
|
||||
elif [ ! -e $KERNEL ]; then
|
||||
echo "$KERNEL not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Writing crash summary to $FILE."
|
||||
|
||||
# Simulate uname
|
||||
ostype=$(echo -e printf '"%s", ostype' | gdb -x /dev/stdin -batch $KERNEL)
|
||||
osrelease=$(echo -e printf '"%s", osrelease' | gdb -x /dev/stdin -batch $KERNEL)
|
||||
version=$(echo -e printf '"%s", version' | gdb -x /dev/stdin -batch $KERNEL | \
|
||||
tr '\t\n' ' ')
|
||||
machine=$(echo -e printf '"%s", machine' | gdb -x /dev/stdin -batch $KERNEL)
|
||||
|
||||
exec > $FILE 2>&1
|
||||
|
||||
echo "$HOSTNAME dumped core - see $VMCORE"
|
||||
echo
|
||||
date
|
||||
echo
|
||||
echo "$ostype $HOSTNAME $osrelease $version $machine"
|
||||
echo
|
||||
sed -ne '/^ Panic String: /{s//panic: /;p;}' $INFO
|
||||
echo
|
||||
|
||||
# XXX: /bin/sh on 7.0+ is broken so we can't simply pipe the commands to
|
||||
# kgdb via stdin and have to use a temporary file instead.
|
||||
file=`mktemp /tmp/crashinfo.XXXXXX`
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "bt" >> $file
|
||||
echo "quit" >> $file
|
||||
kgdb $KERNEL $VMCORE < $file
|
||||
rm -f $file
|
||||
echo
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "ps -axl"
|
||||
echo
|
||||
ps -M $VMCORE -N $KERNEL -axl
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "vmstat -s"
|
||||
echo
|
||||
vmstat -M $VMCORE -N $KERNEL -s
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "vmstat -m"
|
||||
echo
|
||||
vmstat -M $VMCORE -N $KERNEL -m
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "vmstat -z"
|
||||
echo
|
||||
vmstat -M $VMCORE -N $KERNEL -z
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "vmstat -i"
|
||||
echo
|
||||
vmstat -M $VMCORE -N $KERNEL -i
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "pstat -T"
|
||||
echo
|
||||
pstat -M $VMCORE -N $KERNEL -T
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "pstat -s"
|
||||
echo
|
||||
pstat -M $VMCORE -N $KERNEL -s
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "iostat"
|
||||
echo
|
||||
iostat -M $VMCORE -N $KERNEL
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "ipcs -a"
|
||||
echo
|
||||
ipcs -C $VMCORE -N $KERNEL -a
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "ipcs -T"
|
||||
echo
|
||||
ipcs -C $VMCORE -N $KERNEL -T
|
||||
echo
|
||||
|
||||
# XXX: This doesn't actually work in 5.x+
|
||||
if false; then
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "w -dn"
|
||||
echo
|
||||
w -M $VMCORE -N $KERNEL -dn
|
||||
echo
|
||||
fi
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "nfsstat"
|
||||
echo
|
||||
nfsstat -M $VMCORE -N $KERNEL
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "netstat -s"
|
||||
echo
|
||||
netstat -M $VMCORE -N $KERNEL -s
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "netstat -m"
|
||||
echo
|
||||
netstat -M $VMCORE -N $KERNEL -m
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "netstat -id"
|
||||
echo
|
||||
netstat -M $VMCORE -N $KERNEL -id
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "netstat -anr"
|
||||
echo
|
||||
netstat -M $VMCORE -N $KERNEL -anr
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "netstat -anA"
|
||||
echo
|
||||
netstat -M $VMCORE -N $KERNEL -anA
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "netstat -aL"
|
||||
echo
|
||||
netstat -M $VMCORE -N $KERNEL -aL
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "fstat"
|
||||
echo
|
||||
fstat -M $VMCORE -N $KERNEL
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "dmesg"
|
||||
echo
|
||||
dmesg -a -M $VMCORE -N $KERNEL
|
||||
echo
|
||||
|
||||
echo "------------------------------------------------------------------------"
|
||||
echo "kernel config"
|
||||
echo
|
||||
config -x $KERNEL
|
||||
Loading…
Reference in New Issue
Block a user