# Userspace DTrace (USDT) {#usdt} ## Package Dependencies These dependencies are needed for building bpftrace and the sys/sdt.h header file that SPDK libraries will include for DTRACE_PROBE macro definitions. Fedora: libbpf gtest-devel gmock-devel bcc-devel systemtap-sdt-devel llvm-devel bison flex Ubuntu: systemtap-sdt-dev libbpfcc-dev libclang-7-dev bison flex ## Building bpftrace We have found issues with the packaged bpftrace on both Ubuntu 20.04 and Fedora 33. So bpftrace should be built and installed from source. ``` git clone https://github.com/iovisor/bpftrace.git mkdir bpftrace/build cd bpftrace/build cmake -DCMAKE_BUILD_TYPE=Release .. make sudo make install ``` ## bpftrace.sh bpftrace.sh is a helper script that facilitates running bpftrace scripts against a running SPDK application. Here is a typical usage: ``` scripts/bpftrace.sh `pidof spdk_tgt` scripts/bpf/nvmf.bt ``` Attaching to USDT probes requires the full path of the binary in the probe description. SPDK bpftrace scripts can be written with an __EXE__ marker instead of a full path name, and bpftrace.sh will dynamically replace that string with the full path name using information from procfs. It is also useful to filter certain kernel events (such as system calls) based on the PID of the SPDK application. SPDK bpftrace scripts can be written with a __PID__ marker, and bpftrace.sh will dynamically replace that string with the PID provided to the script. ## Configuring SPDK Build ``` ./configure --with-usdt ``` ## Start SPDK application and bpftrace script From first terminal: ``` build/bin/spdk_tgt -m 0xC ``` From second terminal: ``` scripts/bpftrace.sh `pidof spdk_tgt` scripts/bpf/nvmf.bt ``` nvmf.bt will print information about nvmf subsystem and poll group info state transitions. From third terminal: ``` scripts/rpc.py <