45c42ac2f2
This is done in order to track core dumps in a more efficient manner. Till now, some cores could be missed if the binary was executed outside of the cwd of the autotest (i.e. outside of the spdk repo) but which was part of the critical path of the actual test (e.g. fio in vhost-initiator tests). Also, since core_pattern was set to plain "core", impact on the underlying storage wasn't controlled either - if core was 20G in size, this is what we would get. This could easly exhaust storage in case error-prone patchsets were submitted on the CI side. The collector will try to mitigate all the above by doing the following: - collecting all the cores, regardless of their cwd - limiting size of the core to 2G - compressing the cores (lz4) Also, limit of 2 collectors executing at once is set - if more processes crashes at approx. the same time, they will be logged in the kernel log instead. Signed-off-by: Michal Berger <michalx.berger@intel.com> Change-Id: I5956a9030c463ae85a21bfe95f28af5568c5c285 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5369 Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Mellanox Build Bot Reviewed-by: Karol Latecki <karol.latecki@intel.com> Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
90 lines
2.0 KiB
Bash
Executable File
90 lines
2.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# We don't want to tell kernel to include %e or %E since these
|
|
# can include whitespaces or other funny characters, and working
|
|
# with those on the cmdline would be a nightmare. Use procfs for
|
|
# the remaining pieces we want to gather:
|
|
# |$rootdir/scripts/core-collector.sh %P %s %t $output_dir
|
|
|
|
get_rlimit() {
|
|
local limit
|
|
|
|
while read -ra limit; do
|
|
[[ ${limit[1]} == core ]] && echo "${limit[4]}" # soft
|
|
done < "/proc/$core_pid/limits"
|
|
}
|
|
|
|
core_meta() {
|
|
jq . <<- CORE
|
|
{
|
|
"$exe_comm": {
|
|
"ts": "$core_time",
|
|
"size": "$core_size bytes",
|
|
"PID": $core_pid,
|
|
"signal": "$core_sig ($core_sig_name)",
|
|
"path": "$exe_path",
|
|
"statm": "$statm"
|
|
}
|
|
}
|
|
CORE
|
|
}
|
|
|
|
bt() { hash gdb && gdb -batch -ex "thread apply all bt full" "$1" "$2" 2>&1; }
|
|
|
|
stderr() {
|
|
exec 2> "$core.stderr.txt"
|
|
set -x
|
|
}
|
|
|
|
args+=(core_pid)
|
|
args+=(core_sig)
|
|
args+=(core_ts)
|
|
args+=(output_dir)
|
|
|
|
read -r "${args[@]}" <<< "$*"
|
|
|
|
exe_path=$(readlink -f "/proc/$core_pid/exe")
|
|
exe_comm=$(< "/proc/$core_pid/comm")
|
|
statm=$(< "/proc/$core_pid/statm")
|
|
core_time=$(date -d@"$core_ts")
|
|
core_sig_name=$(kill -l "$core_sig")
|
|
|
|
core=$output_dir/${exe_path##*/}_$core_pid.core
|
|
stderr
|
|
|
|
# RLIMIT_CORE is not enforced when core is piped to us. To make
|
|
# sure we won't attempt to overload underlying storage, copy
|
|
# only the reasonable amount of bytes (systemd defaults to 2G
|
|
# so let's follow that). But first, check limits of terminating
|
|
# process to see if we need to make any adjustments.
|
|
max_core=$((1024 * 1024 * 1024 * 2))
|
|
|
|
rlimit=$(get_rlimit)
|
|
if [[ $rlimit == unlimited ]] || ((rlimit > max_core)); then
|
|
rlimit=$max_core
|
|
fi
|
|
|
|
# Nothing to do
|
|
((rlimit == 0)) && exit 0
|
|
|
|
# Clear path for lz
|
|
rm -f "$core"{,.{bin,bt,gz,json}}
|
|
|
|
# Slurp the core
|
|
head -c "$rlimit" <&0 > "$core"
|
|
core_size=$(wc -c < "$core")
|
|
|
|
# Compress it
|
|
gzip -c "$core" > "$core.gz"
|
|
|
|
# Save the binary
|
|
cp "$exe_path" "$core.bin"
|
|
|
|
# Save the backtrace
|
|
bt "$exe_path" "$core" > "$core.bt.txt"
|
|
|
|
# Save the metadata of the core
|
|
core_meta > "$core.json"
|
|
|
|
# Nuke the original core
|
|
rm "$core"
|