Since the trace files can get very large (several GBs), parsing them
using python's json module might require an unfeasible amount of memory,
as it needs to load the whole file first. The ijson [1] library
provides interfaces for parsing files iteratively, only loading a small
portion of a file at a time.
It requires the input JSON to have the tsc_rate and the definitions of
the tracepoints listed before the tracepoint entries. It's not a big
deal, as this is the way `spdk_trace` generates it, but it's worth
noting, as passing that file through something like `jq -S` might make
it unreadable to the trace script.
[1] https://pypi.org/project/ijson
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I03c0c3fb47091da615a3978b8d63edf4d876b811
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8275
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Monica Kenguva <monica.kenguva@intel.com>
This patch adds support for annotating qpairs. The raw pointer values
are replaced with a list of various properties of a qpair collected by
the bpftrace script.
A line like this:
`0 2856282624.000 RDMA_REQ_NEW id: r3256 qpair: 0x26ba550`
becomes (the line was only broken here):
`0 2856282502.669 RDMA_REQ_NEW id: r3256 qpair(ptr=0x26ba550, thread=2, qid=0, \
subnqn=nqn.2016-06.io.spdk:cnode1, \
hostnqn=nqn.2014-08.org.nvmexpress:uuid:729783b4ab38485d8d767b7741108a8)`
To annotate a trace, one first needs to enable and record the DTrace
probes:
```
$ scripts/bpf/trace.py --record `pidof spdk_tgt` > bpftraces
^C
```
Of course, the probe events are only recorded when the script is
executing, so in order to generate the annotations properly, it must be
started before the annotated objects are created. For instance, for
NVMeoF, it needs to be running before a connection is made.
After the BPF probes are recored, the traces can be annotated:
```
$ build/bin/spdk_trace -p `pidof spdk_tgt` -s spdk_tgt -j | \
scripts/bpf/trace.py -b bpftraces
```
For now, the script only annotates traces from the rdma module, as it's
the only one with tracpoints recording qpair pointers now, but it could
be extended to support more tracepoints.
Similarly, more objects could be annotated in the future by extending
the `SPDKObject` class and defining additional DTrace probe points.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I2225f4b1ab266b41af561b5e7d942411cd0c13c1
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8107
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This patch introduces definitions responsible for generating bpftrace
scripts and parsing its output. That output will be used in subsequent
patches to provide annotations for SPDK traces.
The script has a hardcoded set of probe points that are used to generate
the bpftrace script. They're also checked against the probes present in
code to sanitize them and make sure that they're in sync.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I1b8c95e1a035bd7affed2c44b056828a5da94abd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8106
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
This script will use the output generated by the spdk_trace app (JSON) and
replace some of the raw pointers with the information gathered through
the DTrace probes.
For now, this only prints out the traces without doing any annotations,
so the output should be nearly identical to what's printed by
spdk_trace. The annotations will be added in subsequent patches.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I406e8d4fde9cb88b31c38aa96f11b88e607e9436
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8105
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This will allow to match and compare the DTrace probes with SPDK traces
as they'll be using the same source for the timestamps.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I686c3351428c75f9b618a1a909836504fccad828
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8101
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
The traces are tracking the lifecycle of a poll group: creating it,
adding and disconnecting qpairs, and finally destroying the group.
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I075b7f24d14b8fbb42bb18ddd70a668a8bace118
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7158
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Also add scripts/bpf/nvmf.bt to enable and log these
probes.
This patch also adds a script that can generate
a bpftrace script snippet with string maps for
needed enumerations (currently nvmf_tgt_state and
spdk_nvmf_subsystem_state). This allows us to
dynamically generate this from the source code, and
can be extended for other enums we may want to
add in the future.
Thanks to Michal Berger for converting my original
gen_enums.py script into gen_enums.sh!
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Iff34a6218aef40055ac14932eea5fc00e1c8bcf5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7194
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
This script counts how many times each syscall
is called by the SPDK application.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ife5efccee73e407e1d819293786f2acdfa317f78
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7713
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
This script captures a histogram of the returned values
from the readv syscall. It can be useful for analyzing
the efficiency of our sock/posix implementation.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I910eb2e81e3416855cda7a3854982cd02476850e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7712
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
This script tracks spdk_thread_send_msg and
spdk_for_each_channel calls. It also uses usym
to get the name of the function pointer passed to
these calls.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia0cf5b50538bd26a3e15e35cbab765fe3c327c73
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7711
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
