Commit Graph

1464 Commits

Author SHA1 Message Date
Michal Berger
8c176db2ca check_format: Fixes for shellcheck's SC2030 directive
SC2030: Modification of var is local (to subshell caused by pipeline)

This directive is raised since the for loop (or rather its contents)
is piped through to another process. In this context it's not harmful
since we are interested only in sending the stdout but since we operate
on variables that happen to be local to the process that handles the
loop shellcheck sees that as a potential issue. To make it happy
remove the pipe and sort pci addresses prior running the loop.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I835c5ac1da9012129c5d01d62880307f70caab1b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8501
Community-CI: Mellanox Build Bot
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-06-30 22:43:05 +00:00
Michal Berger
36f5f0df17 check_format: Fixes for shellcheck's SC2155 directive
This is a first commit in the series addressing potential Bash issues
as discovered by the latest shellcheck release (0.7.2, shipped with
the very latest fedora33 and fedora34). The goal is to either fix,
locally or globally disable given directive(s).

SC2155: Declare and assign separately to avoid masking return values

Simplify the setting of the variable pointing at the root of the repo.
Also, keep it consistent with the rest of the scripts and declare it as
$rootdir.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I63e0b1a85ce16f7983e9ba6dd985046e8a39a650
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8500
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-06-30 22:43:05 +00:00
Jim Harris
79b6771f61 scripts: add escape in check_json_rpc string
On my system, the git grep for ^SPDK_RPC_REGISTER(
fails because the ( is unmatched.  It seems we should
escape it, although clearly our CI systems don't seem
to require it.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6412e27e79488616743f76b2d9eac8f62b996078

Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8393
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-06-30 19:23:50 +00:00
Konrad Sztyber
b0d3f29a33 scripts/trace: added script to annotate traces
This script will use the output generated by the spdk_trace app (JSON) and
replace some of the raw pointers with the information gathered through
the DTrace probes.

For now, this only prints out the traces without doing any annotations,
so the output should be nearly identical to what's printed by
spdk_trace.  The annotations will be added in subsequent patches.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I406e8d4fde9cb88b31c38aa96f11b88e607e9436
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8105
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-06-28 16:17:35 +00:00
Konrad Sztyber
a4044e77d6 usdt: pass ticks as the first probe argument
This will allow to match and compare the DTrace probes with SPDK traces
as they'll be using the same source for the timestamps.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I686c3351428c75f9b618a1a909836504fccad828
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8101
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
2021-06-28 16:17:35 +00:00
yidong0635
928db41626 rpc: Fix raid strip size method not consistent.
Fix json config file using strip_size error, and
The strip_size parameter was changed to strip_size_kb  in `bdev_raid_create`
     rpc function in order to make it more meaningful to users.

Unify the coding style in rpc help file.
      using --strip-size-kb --raid-level ... instead of --strip-size_kb.

Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: I341af1034c7ea6716ff839bb524ee733c9e0cd34
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8401
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-06-24 17:30:19 +00:00
Shuhei Matsumoto
11d76e8c81 iscsi: Set resource pool size to the predetermined values to run with little memory
Add three parameters, pdu_pool_size, immediate_data_pool_size, and
data_out_pool_size to the RPC iscsi_set_options to run iSCSI target
with little memory.

For some use cases, we want to keep the max number of connections,
but simultaneously we want to reduce the pool size and let I/Os wait
until resource is provided.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I74dc785310b1d985f3e338c1e13fba3a3840d113
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8191
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-06-17 08:59:18 +00:00
Michal Berger
1347830beb scripts/vagrant: Allow to mix CMB and PMR setup
As of latest qemu (>= 6.0.0) these can now be used together as
different BARs are used to accommodate for both.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Icf32f491fd33ef733f0d946f9b5caaeb87137d28
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8217
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-06-14 08:43:01 +00:00
Michal Berger
a5c3f72a89 scripts/vagrant: Initialize pmr_cmdline as an empty string
Having pmr_cmdline set to nil was for some reason breaking cmb setup
as weirdly enough pmr_cmdline was ending up being set to "true".
Avoid this by having pmr_cmdline set to some value, empty string in
this case.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Ib41da699dcc2f213f0b18d21be3fd32f398bae41
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8216
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-06-14 08:43:01 +00:00
Konrad Sztyber
a266b6e4be nvmf: add traces around poll group lifecycle
The traces are tracking the lifecycle of a poll group: creating it,
adding and disconnecting qpairs, and finally destroying the group.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I075b7f24d14b8fbb42bb18ddd70a668a8bace118
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7158
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-06-10 10:58:05 +00:00
Jim Harris
3a0f624499 nvmf: add USDT probes for subsystem state changes
Also add scripts/bpf/nvmf.bt to enable and log these
probes.

This patch also adds a script that can generate
a bpftrace script snippet with string maps for
needed enumerations (currently nvmf_tgt_state and
spdk_nvmf_subsystem_state).  This allows us to
dynamically generate this from the source code, and
can be extended for other enums we may want to
add in the future.

Thanks to Michal Berger for converting my original
gen_enums.py script into gen_enums.sh!

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Iff34a6218aef40055ac14932eea5fc00e1c8bcf5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7194
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2021-06-08 08:27:15 +00:00
Nick Connolly
5a21edf4e3 scripts: add autotest support for Windows
Source Windows specific definitions from common.sh. These adjust
uname to report the operating system as WSL, Msys or Cygwin and the
kernel name as Windows.  They also define kill() to invoke the SIGTERM
handler before causing a hard stop with TerminateProcess.

Adjust the ordering of the 'source' commands in autotest_common.sh so
that the config definitions are available when common.sh is loaded.
Define MAKE, MAKEFLAGS and HUGEMEM for Windows.

Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Change-Id: I130b892ee55c925a0b033bda271a29133993afb7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7101
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2021-06-08 08:19:32 +00:00
jiaqizho
81a8bc670c iostat : add -x to display extended statistics
Signed-off-by: jiaqizho <jiaqi.zhou@intel.com>
Change-Id: Ia25257093a13987c40a682c9b5ace4daa15ac607
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4847
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-06-04 22:47:44 +00:00
yupeng
b832f99f88 nvmf: Add nvmf_set_crdt RPC
Set the three CRDT values at SPDK_RPC_STARTUP time.

Signed-off-by: Peng Yu <yupeng0921@gmail.com>
Change-Id: I2fb4c4a3e367a4888cfec4658e6bf6899c7ae1f4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8007
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-06-04 20:02:42 +00:00
Karol Latecki
4e4c2cd8e3 scripts/nvmf_perf: make rate_iops actually optional
Change-Id: Ie46df6e4f52b59941a90eeb786954fe7d5f1d089
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8040
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Michal Berger <michalx.berger@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-05-28 14:25:54 +00:00
Michal Berger
43aca60458 scripts/fio-wrapper: Move devices sync to configure_devices()
This is to make sure we sync devices for both nvmf and iscsi
protocols.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Ifec6cc2d240a5b22b73e9f4087ae328568e4fe4a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8042
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
2021-05-27 07:35:29 +00:00
Ziye Yang
c3cf9ec295 rbd/rpc: Add a rpc call to get the info of the clusters.
Purpose: Let the users know the current available registered Rados
cluster and the related info.

Change-Id: I115c129ae6e4b0372579aad168fd88f8be136357
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7990
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-05-26 09:21:48 +00:00
Michal Berger
6c1a1a3dca scripts/rpc: Make sure address argument is properly interpreted
In case the addr argument was not an existing unix socket file the rpc
client would consider it to be an actual ip address. As a result
connect() would be called with improper set of arguments. This could
cause the rpc.py to block for undesired amount of time until connect()
finally decided to return (seen on some fedora33 builds).

This was affecting sh wrapper functions like waitforlisten() which
use rpc.py to determine if given app is ready to be talk to blocking
execution of the tests for way too long then intendent.

To avoid such a scenario determine the format of the address and use
routines proper for given address family.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Iaac701d72c772629fa7c6478ff4781b0c5d485d5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7777
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
2021-05-24 10:11:05 +00:00
Michal Berger
7bea0a074f scripts/vagrant: Create user netdev for openstack network
The openstack tests require to have one interface available under
specific ip, 10.0.2.15. This ip is a first lease VM receives from
the user network which is preconfigured in libvirt environment.

Use this instead of defining completely separate network which
by the very default will serve its own DHCP service as well. This
may result in a cosmetic issues (e.g. two ips, static one and
dynamic one on the interface) of having inconsistent config on the
VM depending on how the net device is configured by the underlying
distribution.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Ie3da26eebcfb0de668166c4a0120c65aec104540
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7441
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
2021-05-24 09:58:17 +00:00
Karol Latecki
b686c82581 scripts/nvmf_perf: update add_listener rpc call
nvmf_subsystem_add_listener() in rpc/nvmf was modified
by 64debe0453 and the
change was not reflected in performance scripts.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I7bb8c39ab225421964e37883aa7b0f824e8df683
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7978
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2021-05-21 08:04:24 +00:00
Jim Harris
887109bf9e scripts: add bpf/syscalls.bt
This script counts how many times each syscall
is called by the SPDK application.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ife5efccee73e407e1d819293786f2acdfa317f78
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7713
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2021-05-21 08:03:27 +00:00
Jim Harris
dca7ef2759 scripts: add bpf/readv.bt
This script captures a histogram of the returned values
from the readv syscall.  It can be useful for analyzing
the efficiency of our sock/posix implementation.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I910eb2e81e3416855cda7a3854982cd02476850e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7712
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2021-05-21 08:03:27 +00:00
Jim Harris
cbb57e0d96 scripts: add bpf/send_msg.bt
This script tracks spdk_thread_send_msg and
spdk_for_each_channel calls.  It also uses usym
to get the name of the function pointer passed to
these calls.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia0cf5b50538bd26a3e15e35cbab765fe3c327c73
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7711
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2021-05-21 08:03:27 +00:00
Jim Harris
af0bf0f006 scripts: add bpftrace.sh
This script simplifies execution of bpftrace scripts
for SPDK application analysis. The script should be
invoked as:

scripts/bpftrace.sh `pidof spdk_tgt` script.bt

where 'script.bt' is the name of the bpftrace script
you wish to execute.

The script helps simplify the following:

1) replace __EXE__ markers in the bpftrace script with
   the full path of the spdk executable (required for
   usdt probes);  it uses /proc/<pid>/exe to get the
   full path
2) replace __PID__ markers in the bpftrace script with
   the PID of the spdk process under analysis - this
   allows for doing things like filtering system call
   counts for just the SPDK process
3) invoke bpftrace with the -p <pid> option

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib271f26efe4ba326d951a7b30f61b22be755dda7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7710
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-05-21 08:03:27 +00:00
Jim Harris
5a1c74bfa4 usdt: add User Space DTrace support to SPDK
For now, we will keep this disabled by default,
enable with --with-usdt option to the configure
script.  Long-term we will want to enable this by
default, and only disable via configure.

Modules can include spdk_internal/usdt.h and add
probes such as:

SPDK_DTRACE_PROBE2(probe_name, ptr, val);

When USDT is enabled, these will translate to
DTRACE_PROBE2(spdk, probe_name, ptr, val).  When
USDT is disabled, these will translate to nothing.

Later patches will add some probe points to the
nvmf target, some bpftrace scripts, and instructions
for how to successfully capture data with these
probe points and scripts.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id168e2c800fa5522815a175026386319014cfdaa
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7173
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2021-05-21 08:03:27 +00:00
Michal Berger
9b3773db68 scripts/common: Cache pci modalias
Additionally, include the first resolved alias of the module.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I71c7c15398ea36f3dea3340f5c98568ff6b6a68a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7848
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-05-20 15:16:21 +00:00
Michal Berger
5f61a0c919 scripts/common: Cache the pci driver
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Ie0ab6b31757b0446c697f59bfc789e78207b6e34
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7847
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-05-20 15:16:21 +00:00
Tomasz Zawadzki
1c71de70bd perf/nvmf: add FIO I/O rate limit to the config
Added 'rate_iops' FIO option to the performance scripts.

This will be used during tests where limiting factor
has to be artifically imposed, rather than come from
the target/initiator system or enviroment.

One case for this are scheduler tests, to limit
each connection and balance the threads accordingly.

Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I10bb09be9205a8b7ba90b36cd4e0bf922b0ecbc3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7840
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-05-20 15:14:59 +00:00
Ziye Yang
5c0160263c bdev/rbd: Revise bdev_rbd_create rpc function.
Revise bdev_rbd_create rpc call to add an optional
parameter "--cluster-name", e.g., "--cluster-name Rados".

Then users can create a rbd bdev with registered
Rados Cluster. This shared strategy can be used to
remove the thread creation overhead if multiple rbds
are connected to the same Ceph cluster.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: Ide5800f8fc6b2074805272a59731c666fe279b9a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7584
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-05-19 08:08:39 +00:00
Ziye Yang
e18eaee2af bdev/rbd: Add cluster register/unregister RPC support
This patch is used to add two rpc calls:
bdev_rbd_register_cluster
bdev_rbd_unregister_cluster

Then in the next patch, rbd bdev constructed on the same cluster object
can share the common Rados_t structure in order to remove the thread creation
overhead and improve the scalability.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I898cc4ffabb8e6721ba5bef099cbf948c64d2c98
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7551
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-05-17 09:52:18 +00:00
Konrad Sztyber
4f11fa5b6c scripts/setup: use test -e to test for msr presence
Current `test -f` never succeeds, because `/dev/cpu/0/msr` is a
character device file.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iaa28f04866ab24140668ad43351a37f0ccccef65
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7166
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-05-14 10:06:41 +00:00
Michal Berger
3c578739e9 scripts/fio-wrapper: Make sure all nvme block devices exist
This is needed before we can start looking up attributes
of the block subsystem under sysfs. In some occasions they
may simply not exist yet at a given time.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I4261d17e92e02acec31b8affaccd0aa11c2b851e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7863
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-05-13 10:09:03 +00:00
Jonathan Teh
be6a01efb3 nvmf: Allow configurable controller ID range
Add {min,max}_cntlid to spdk_nvmf_subsystem, defaulting to 1 and
0xFFEF, respectively, and add nvmf_subsystem_set_cntlid_range() to
allow the controller range to be configured in the range [min_cntlid,
max_cntlid].

Also add {min,max}_cntlid to the nvmf_create_subsystem RPC to allow
the controller ID range to be specified when creating an nvmf
subsystem.

Signed-off-by: Jonathan Teh <jonathan.teh@mayadata.io>
Change-Id: I936db3bb0c9a38569063a6fd3c11df262dfad776
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7322
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2021-05-12 09:18:48 +00:00
Michal Berger
b96ec35dc8 scripts: Remove fio.py
With https://review.spdk.io/gerrit/c/spdk/spdk/+/7684 there are no
more active uses of it in the test suites.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: If6f8227b68f1fe1a4038cec2b6107eac82ad44de
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7698
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-05-12 08:48:10 +00:00
Michal Berger
19f0c9a0d0 autotest: Replace fio.py with a bash wrapper in tests
fio.py simply wraps itself around fio and doesn't do anything that
would require python to be in use. Having it in a simple bash form
makes it easier to integrate it with autotest's common sh tooling
and to debug any potential issues with the underlying tests.

This also fixes #1919 by making sure only proper nvme devices are
selected for the nvmf targets.

Fixes: #1919.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I111d00df3c7b2517f431cae865e258a665c2ecb3
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7684
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-05-12 08:48:10 +00:00
jiaqizho
64debe0453 rpc: add a command parser
The changes in the nvmf_create_transport show
how this command parser work. And there have
two benefit for this changed.

1. Simplify the definition of rpc method. no need add
so many args anymore. Also it retains its original
functions, so we can also check the input args.

2. Make the rpc call more versatile, for example.
when user extend the subparsers(add new args
into subparsers), they can pass some private args
into the rpc method by command parser.

Signed-off-by: jiaqizho <jiaqi.zhou@intel.com>
Change-Id: Iaf916e3454f23715cf9216794bb80c65b2b4603f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6652
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2021-05-11 12:02:00 +00:00
Michal Berger
eb4b79919a pkgdep/git: Bump vanilla qemu to v6.0.0
Older version wasn't providing full support for PMR setup, this one
does.

Rewrite get-pmr such that it supports CMB/PMR sizes greater than 4GB.
Also, since CMB and PMR can coexist in newer versions of qemu try to
detect both under single device.

E.g.:

nvme0:0000:00:04.0:64-bit:prefetchable:0x700000000:0x707ffffff:0x08000000:cmb
nvme0:0000:00:04.0:64-bit:prefetchable:0x400000000:0x5ffffffff:0x200000000:pmr
nvme1:0000:00:05.0:64-bit:prefetchable:0x708000000:0x70fffffff:0x08000000:cmb
nvme2:0000:00:06.0:64-bit:prefetchable:0x600000000:0x6ffffffff:0x100000000:cmb

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: Ic159f5c12d3ef39db77617f7d64f825356c255a4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7539
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-05-11 11:51:01 +00:00
Michal Berger
d97b106f81 pkgdep: Update liburing version to 2.0 release
Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I239c6eb2e9919a4605a14d39f4462b3aa5f54883
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7496
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Monica Kenguva <monica.kenguva@intel.com>
2021-05-05 13:27:16 +00:00
John Kariuki
b69796f777 scripts/perf: fix json param names
Fixed parameter names in the sample json config file.

Change-Id: I4e221b17c533c0957a325e24043a627b5acb6220
Signed-off-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7688
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-05-05 13:25:48 +00:00
Alexey Marchuk
2fd97e28bf sock: Deprecate enable_zerocopy_send in sock_impl_set_options RPC
This deprecated parameter will be removed in SPDK 21.07

Change-Id: I2b2fbcc798bb50fa6f9dfe35045f66e41c1ceaa9
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7608
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-04-27 08:13:32 +00:00
Alexey Marchuk
8e85b675fc sock: Add new params to configure zcopy for server, client sockets
When zcero copy send is enabled and used by initiator,
it could significantly increase latency in some payloads.
To enable more fine graing configuration of zero copy
send feature, add new parameters enable_zerocopy_send_server
and enable_zerocopy_send_client to spdk_sock_impl_opts to
enable/disable zcopy for specific type of sockets.
Exisiting enable_zerocopy_send parameter affects all types
of sockets.

Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I111c75608f8826980a56e210c076ab8ff16ddbdc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7457
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-04-27 08:13:32 +00:00
Ben Walker
03a3a5c02d scripts: Add check_format test for JSON-RPC documentation
All JSON-RPCs must be documented in jsonrpc.md

Change-Id: Iec9119a01433fd0cbe4df4477ce94cc2feeb0d54
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4074
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Monica Kenguva <monica.kenguva@intel.com>
Community-CI: Mellanox Build Bot
2021-04-26 06:55:03 +00:00
Maciej Wawryk
4503500e49 scripts/perf: fix measure_bandwidth failed to open outfile error
Signed-off-by: Maciej Wawryk <maciejx.wawryk@intel.com>
Change-Id: I9b9f8c9e407425fdab1e84050354481ad76c19c2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6655
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
2021-04-22 15:53:34 +00:00
Karol Latecki
9bacb8629d scripts/nvmf_perf: rename remote_nic_ips
Rename to target_nic_ip to better match NVMe-oF
terminology.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I0644410cfb063eeea740c54f5860aeefdb52d52d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7089
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-04-21 23:59:40 +00:00
Karol Latecki
ad7833591c scripts/nvmf_perf: copy SPDK from target early
SPDK directory needs to be copied to initiator systems
earlier for set_irq_affinity to run.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I0f9ade02e58c9224e0b2ddc75117baaf653be614
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6776
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-04-21 23:59:40 +00:00
Karol Latecki
02cb1fe86a scripts/nvmf_perf: allow ADQ for SPDK mode only
Need more investigating how to properly enable ADQ
for Kernel mode. Do not run Kernel + ADQ configuration
for now.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I700ef417b9b398c067a586a1a5ad9947a92057ad
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6566
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-04-21 23:59:40 +00:00
Karol Latecki
b06212cfb5 scripts/nvmf_perf: re-write nvmf perf readme file
Readme was frequently missed when adding new stuff or
updating the performance script, and it is out of date.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I1710473be576ffbcc2fa8e3701b196bd46cf6654
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6538
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-04-21 23:59:40 +00:00
Karol Latecki
c0fc19f36d scripts/nvmf_perf: configure adq traffic classes
Create a default traffic class (tc0) with minimum
needed number of assigned threads and a priority class
(tc1) with number of assigned threads equal to
number of application threads.
Finally run set_xps_rxqs to configure symmertic queues.

set_xps_rxqs script used from Intel ICE driver
package available at:
https://downloadcenter.intel.com/download/29746/
Intel-Network-Adapter-Driver-for-E810-Series-Devices-under-Linux-

Change-Id: Ie0f2db266621a9dabb1621344bfdc5fa64fee03c
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6537
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
2021-04-21 23:59:40 +00:00
Karol Latecki
42d2e588d6 scripts/nvmf_perf: Keep track of num_cores for Initiator class
There are multiple cases for counting number of CPU threads
on Initiator side. Keep track of num_cores in all of them.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: I4b8d1d79a2aa17706d206b149c1c8e8816ae2ef5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6536
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
2021-04-21 23:59:40 +00:00
Karol Latecki
31c50e110c scripts/nvmf_perf: disable sysctl busy_read timeout
Set busy_poll to 0 for non-ADQ. Having it enabled with
this value results is huge latency increase.

Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Change-Id: Ic68ed2b7dc19b8d4b274ce8622194489693863cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7480
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-04-21 23:59:40 +00:00