Commit Graph

341 Commits

Author SHA1 Message Date
Jim Harris
199fa6015d util: add zipf random number generator
zipf is a power law probability distribution. When
applied to performance testing of block devices, it
will select blocks over the full range of LBAs, but
will more frequently select lower-numbered LBAs.
The theta parameter governs the distribution - higher
values of theta will concentrate the distribution on
a smaller number of LBAs.

Note that fio supports zipf, so adding it to SPDK
will enable our perf tools (bdevperf, nvme-perf) to
provide similar functionality.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7df129c9d61996a2070188c6cd9f1fde631ac208
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7779
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-05-17 11:29:52 +00:00
Jim Harris
b549f4113c build: relink apps when vfio-user library is updated
Look for 'vfio_user' in SPDK_LIB_FILES to determine
whether the target application actually needs to be
rebuilt.

Fixes issue #1930.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If3f699c91972e20dc4520e25c655ed11f6843be0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7826
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-05-11 12:04:36 +00:00
Jim Harris
1f5bae841a build: also look for vfio-user libraries in 'lib'
At least on some Ubuntu distributions, default
vfio-user install puts libs in usr/local/lib instead
of usr/local/lib64.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib312d8ca9b4ce9e858a9a9e76525fc8afd48ced5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7825
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: <dongx.yi@intel.com>
2021-05-11 12:04:36 +00:00
Tomasz Zawadzki
47c4304d83 subsystem/nvmf: remove cpu_mask hint when creating poll group
Passing cpu_mask hints that match only single core were
usefull to prevent any accidents when doing round-robin
in case of 'static' scheduler.
In practice this is not required in case of 'static' scheduler,
the threads will be spread out over all reactors anyway.

This hinders other schedulers which try to respect the cpu_mask
hints, as they would not move the thread to any reactor.
Preventing bunching up less used threads on single reactor.

Drawback of this patch is that poll group names will not match
the cores they are on.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5fb308362dd045228ea9fcca24f988388854c054
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7028
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2021-05-11 11:49:52 +00:00
Karol Latecki
7366e569fb build/fio: disable warning for fio clang-11 builds
Fio 3.19 fails to build with Clang 11 because of
implicit-const-int-float-conversion error.
Disable -Werror checking for now.

Fixes #1848

Change-Id: I0f3922d7af412de35d9f79f3f5413af146996bf3
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6991
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2021-04-08 16:09:00 +00:00
root
de8f3a50ee mk: Fix debug build error on ARM ThunderX2 and neoverse N1 platform
When building spdk on ARM platform like thunderx2 with --enable-debug,
there are following error:

/tmp/ccOBb4AF.s: Assembler messages:
/tmp/ccOBb4AF.s:45: Error: selected processor does not support `casp x0,x1,x2,x3,[x4]'
/tmp/ccOBb4AF.s:77: Error: selected processor does not support `caspa x0,x1,x2,x3,[x4]'
/tmp/ccOBb4AF.s:109: Error: selected processor does not support `caspl x0,x1,x2,x3,[x4]'
/tmp/ccOBb4AF.s:141: Error: selected processor does not support `caspal x0,x1,x2,x3,[x4]'

The reason is that DPDK is built with -march=armv8.1-a or -march=armv8.2-a+lse which
have these instructions while SPDK is built with -march=armv8-a+crc which does not support
them. Change spdk build machine to native can fix this.

Signed-off-by: Rui Chang <rui.chang@arm.com>
Change-Id: I759d4ce2c557ce5ff73a802d7a4b6579c4ba64f7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7025
Community-CI: Mellanox Build Bot
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2021-03-25 08:29:09 +00:00
Ziye Yang
d515bf739d bdev/nvme: Use the accelerated_engine
This patch really uses the accelrated engine.
Currently, we only offload the crc32c caculation,
but it can be extended.

Change-Id: If0e4c6a44b6e1e10e03f7eca355bed418d67326b
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6760
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2021-03-23 10:09:39 +00:00
Michal Berger
9239ed33f6 mk: Don't use mime-type to determine type of the lib file
Some versions of the file (e.g. 5.33) confuse the sharedlib files with
pie-executable ones when they already have the executable bit set.
Since this particular bit is already set on the .so.* when they are
built, file may report the type as "application/x-pie-executable"
instead. As a result, lib files won't have proper permissions set
when installed.

This is a bit problematic for building the RPMs since when
find-provides kicks in to determine the dependencies, said files
are not picked up at all - package ends up with empty "provides:"
data breaking the dependency lookup during installation of other
packages which may depend on the broken one.

To mitigate, don't use mime-type, simply run file against the lib
in default mode and look for "shared object" string - this should
work for most of the file versions available on the supported
distribution.

Spotted on Fedora31.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: If9974f5acbdad6eb1204b64f0ee1ac9b2e20f85b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6703
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
2021-03-23 10:08:51 +00:00
Michal Berger
f58afe93b8 test/common: Don't enable werror on FreeBSD
Facing similar issue as with 9ab2855d - can't build SPDK against fio
3.19.

Signed-off-by: Michal Berger <michalx.berger@intel.com>
Change-Id: I4ea22e4f068a19d8d87be836454ec8deeeb2d48a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6805
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-03-18 13:24:50 +00:00
Jacek Kalwas
6e2c2306f9 build: install generated pkg-config files
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I191ad5e3b153fb563256eba1aa695716f66db788
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6377
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2021-03-17 11:10:45 +00:00
Nick Connolly
a4e1d715f7 mk: support file extension for executables
Some platforms require that a specific file extension is used for
executables. Append $(EXEEXT) to $(APP) definitions to accommodate
this. The name EXEEXT has been chosen for consistency with automake
which uses the same variable for the same purpose.

Add extensions for generated files used by Windows to the 'clean' target
(.obj, .lib, .pdb) and add output files to .gitignore.

Tested using a cross compiler that forces a .exe suffix and verifying
that all LINK messages include the correct suffix and that a second
call to make does nothing (showing that the targets match the
generated executables).

Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Change-Id: Ief012f6294d2f94b23c8b4f9747e21a0dfb91e96
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6494
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2021-03-15 09:29:10 +00:00
Nick Connolly
9854c138f7 mk: add support for mingw builds
MinGW builds require a thin layer above the standard libraries to
provide POSIX functionality that is missing on Windows. Add support
for building this.

MinGW cross builds are experimental and work is ongoing to integrate
them into the CI and test environment. Including the changes at this
stage is being done to facilitate that process.

The layer has been arranged in the same way as DPDK and is
accessed as an external build using ./configure --with-wpdk=<dir>.
Support has also been added for using a default ./wpdk in
preparation for reaching the required level of stability.

The help text for ./configure indicates that support for --with-wpdk
is experimental.

Further details and instructions can be found at https://wpdk.github.io.

Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Change-Id: Iff0f705789f19fb193dcb3c9090c3e90613a8d9a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6589
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2021-03-15 08:55:11 +00:00
Nick Connolly
2a8065b554 mk: add mocking for mingw cross builds
On Windows there has to be a thin layer above the system calls to
provide POSIX functionality. When wrapping system calls, the name needs
to be prefixed with wpdk_ to ensure that the layer is called.

Re-arrange the definitions to facilitate wrapping and prefix with wpdk_
for mingw. For other compilers targeting Windows, the --wrap flag is not
supported and the layer above the system calls implements an alternative
mechanism to enable the mocking.

Tested with $(info LDFLAGS) added to spdk.mock.unittest.mk combined with
visual inspection and running the unit tests.

Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Change-Id: If00504740229362c40bee4ae171d35489afb2e77
Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6577
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-03-03 08:16:54 +00:00
Nick Connolly
3399171388 mk: workaround for mingw stack-protector bug
Add -mstack-protector-guard=global to work around GCC bug 86832,
where -fstack-protector will attempt to use native TLS, but mingw only
supports emulated TLS. This causes a segmentation violation at the start
of any function protected by stack-protector. The issue exists in GCC
versions prior to 8.3.

Enable -mstack-protector-guard=global which uses a global variable for
the guard value. This is the behaviour with the fix, so there is no
downside to enabling it for all versions of GCC on Windows.

Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Change-Id: I85bc831043585071b0d8b0a3dd637b8d9ff26dab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6575
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2021-03-03 08:16:54 +00:00
Nick Connolly
af17c1d44d mk: adjust options for mingw cross compilation
Check compiler target triplet for the strings 'windows' or 'mingw'
and set OS to Windows. If found, adjust basic parameters:

Set EXEEXT to indicate executables have .exe suffix.
Exclude 'relro' and 'now' flags which are not recognised on Windows.
Exclude 'noexecstack' flag which is not recognised on Windows.
Exclude -pthread flag because Windows doesn't have POSIX threads.
Include libssp.a to support stack-protector and _FORTIFY_SOURCE.

Signed-off-by: Nick Connolly <nick.connolly@mayadata.io>
Change-Id: I94a45d3123ebc81f5529006fa07b05da897e5866
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6574
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2021-03-03 08:16:54 +00:00
Changpeng Liu
d0b2448b63 mk/libvfio-user: support --with-vfio-user=/usr/path
Users may not use SPDK libvfio-user submodule, so add a
library path for this case, users can still just use
'--with-vfio-user' without adding the path, for this case
a submodule default path will be used.

Change-Id: Ib0dd82ab6910056ff21b5b2d373c0d16916162c6
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6471
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Thanos Makatos <thanos.makatos@nutanix.com>
2021-02-26 12:27:06 +00:00
Changpeng Liu
ae9f80354a mk/libvfio-user: use the install directory for the link path
Previously the libvfio-user didn't install the static library to
the install directory, with that fix in libvfio-user, we can use
the install directory now.

Change-Id: I3f232395a6dc3bf6d7d3937b245eb10c4b6a0e26
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6470
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-02-26 12:27:06 +00:00
Alexey Marchuk
13381936af make: Correct ppc detection
gcc -dumpmachine may provide the following output:
ppc64le-redhat-linux
To detect PowerPC system we should check for both
"powerpc%" and "ppc%" strings

Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: Ifb5afe743797177afa4f1b541a0c8ccaf4befbda
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6378
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: Mellanox Build Bot
2021-02-22 10:04:14 +00:00
Jacek Kalwas
0e0805b551 build: add missing log dep for env_dpdk
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: Ieab3a3851eea9c6c9db2ddbf74d59b17128c3f0f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6403
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
2021-02-17 10:19:06 +00:00
Jim Harris
91e422572a configure: add --enable-cet to build with CET support
Intel Control-flow Enforcement Technology (CET) is a
processor feature that blocks return/jump-oriented
programming (ROP) attacks.

It is currently only supported on Tiger Lake client
processors, but will be available on other processors
in the future.

CET requires toolchain support.  gcc8 does support it.
For now, this will opt-in only at least until the
technology is available on server processors.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8c7f882eeeaed26484c31dc0d67d5cc42baeaa2d

Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5921
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-01-21 19:24:10 +00:00
Jim Harris
79f9a7f572 build: generate pkg-config files for SPDK
Users can now generate the necessary linker args for their
own applications using something like:

PKG_CONFIG_PATH=build/lib/pkgconfig pkg-config --libs spdk_nvme

Dependencies between libraries are included in the generated
.pc files, so the user only needs to pass the top-level subsystems
or individual SPDK libraries they are using in their application.

Modules will automatically be added to the output if the associated
library is specified.  For example, specifying "spdk_bdev" will include
the libraries not only for spdk_bdev, but also all of the bdev modules.

Users still need to supply the -Wl,--no-as-needed or -Wl,--whole-archive
flags. They cannot be added to the .pc files without increasing the length
of the argument string by a factor of 15x to 20x.

Modify the test/external_code/hello_world Makefile to use pkg-config to
ensure this gets tested at some level in our autotest environment.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie48a75f11969d5d775d514cf10bcb82d197eabfd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4371
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-01-21 09:46:32 +00:00
Changpeng
3ff9c13614 NVMe/vfio-user: add initial version vfio-user transport to NVMe driver
When NVMf target linked with vfio-user library, we can use
vfio-user client library to connect to the target.

Here is the three examples that can work with target:

identify -r 'trtype:VFIOUSER traddr:/var/run/muser/domain/muser0/8' -g
perf -r 'trtype:VFIOUSER traddr:/var/run/muser/domain/muser0/8' -g -q 1 -o 4096 -w read -t 10
reconnect -r 'trtype:VFIOUSER traddr:/var/run/muser/domain/muser0/8' -g -q 32 -o 4096 -w randrw \
-M 50 -t 10 -c 0xE

You can run the following test script test/nvmf/target/nvmf_vfio_user.sh to have a quick test,
currently enabled with NVMe Identify,Perf,Reconnect tools.

Change-Id: Ieb9842b2f372184fffbf7f23e4aad26feb47c350
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3839
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-01-21 05:00:18 +00:00
Changpeng Liu
b30d57cdad libvfio-user: include libvfio-user as a submodule with SPDK
Also add a CONFIG_VFIO_USER config flag, it's enabled by
default.

Change-Id: I18b44c024a264516a60f743d5c366a4c7f7c6785
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5000
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2021-01-21 05:00:18 +00:00
Liu Xiaodong
76c21d4138 intr_tgt: interruptable libs shown by modules.mk
Indicate interruptable module libaries by spdk.modules.mk
through INTR_BLOCKDEV_MODULES_LIST.
Other applications can directly use INTR_BLOCKDEV_MODULES_LIST
to create themselves to be interruptable.

Change-Id: Id1894dd3753c7b404ebb9d77f3184be942f6d216
Signed-off-by: Liu Xiaodong <xiaodong.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5782
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-01-15 08:31:42 +00:00
Ziye Yang
d478edd4ee lib/nvmf: Use the async function to caculate the crc32 for header_digest.
In this patch, we will use the accel library to do the crc32 work for the
header digest when sending the pdu in the target side.

For data digest support, will consider in the further patch.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I3169a158afd633f48bdbeb2cce1ed20e4141ae45
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5472
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2021-01-15 08:30:34 +00:00
Jim Harris
400c3b8a52 build: initialize SYS_LIBS in spdk.common.mk
There are cases where spdk.common.mk can get included
more than once in a Makefile.  So avoid duplication
in SYS_LIBS by initializing it before appending text
to it.

The duplication was harmless but a bit annoying.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idf081f973e15a9ab6ec9211b9a3cbf02c736c7a0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5676
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-12-28 13:32:06 +00:00
Jim Harris
8f92bde43e build: add PRIVATE_LIBS variables for module lists
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8e7f8af56b494a9c578cd0eeb718fe6a42deb542
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4469
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-12-28 13:32:06 +00:00
Jim Harris
ec02743a4c build: add ufc to SYS_LIBS in spdk.common.mk
This avoids duplicating it in several different
Makefiles.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I85f4886ec6744c23639d24867e2c68757dfeba32
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4484
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
2020-12-28 13:32:06 +00:00
Jim Harris
d132ee3531 build: use DEPDIRS variables to build SPDK_LIB_LIST
All of our Makefiles duplicate huge lists of libraries
in SPDK_LIB_LIST.  We have a very precise and accurate
accounting of the library dependencies in
mk/spdk.lib_deps.mk which can be used to generate
the full list if the app specifies the modules and
subsystem libraries it wishes to link.

I did a first pass through all of the existing
Makefiles to take advantage of this new functionality.
There may be more optimizations we can make later but
don't want to hold up this patch for all of them.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Icdaf6f749a6908df2c2ce2db22631a4af4ff3a9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5553
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-12-18 09:40:01 +00:00
Jim Harris
895ad89214 ioat: remove whitelist/blacklist functionality
This can be done via explicit options on the
application command line instead.

This has the added benefit of removing a usage of
whitelist/blacklist which we are working to
remove from the SPDK repository.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5739e281f1c29fc8a5d175f5bbc916cd7d926fe4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5274
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-12-03 09:41:07 +00:00
Jim Harris
1b56233c0c build: add SPDK_NO_LIB_DEPS to build libraries w/o dependencies
Our check_so_deps.sh test script removes the spdk.lib_deps.mk
include from spdk.lib.mk so that we can check which symbols
the libraries depend on.  But modifying the code like this
is a bit kludgy.  So instead add a Makefile variable that
check_so_deps.sh can set to get the same behavior.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5622f4c3adb2d5ccd5ae33cb4cd116716134a9b7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4512
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: <dongx.yi@intel.com>
2020-11-13 05:29:38 +00:00
Jim Harris
c94c546e00 build: force shared library dependencies
We use some implicit depdendencies in our subsystem
libraries.  With at least some linkers, these
dependencies are not written into the shared library
file if there is no actual symbol dependency found.
So use --no-as-needed to ensure the dependencies
stick.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idb476efb50b00ff04f5640b4e5a1362a1f096fa7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4796
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: <dongx.yi@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Maciej Wawryk <maciejx.wawryk@intel.com>
2020-11-02 11:23:21 +00:00
paul luse
17bb748a60 accel: Move non-engine specific batch to the accel_fw layer
The new design:

* Supports a generic batching capability in the accel_fw layer
that keeps track of hw accelerated vs sw commands based on
the capabilities of the engine and processes sw commands in the
generic layer while sending a list of commands (not a batch)
to the engines for processing.

* Batch completions are managed via the generic layer, when using
the accel_fw the engines only process commands. With DSA however,
if a list of commands is sent down it will use the DSA public
API to create and send batches but will rely on the generic layer
to complete the batch task itself. When using DSA directly, batching
works as usual (DSA handles batch completion).

* The engine function tables were greatly simplified by replacing
all of the individual entries (copy, fill, crc32c, etc) with one
`submit_tasks` function that is used to both send lists of tasks
for batches or just one task for single shot API.

* Internally batching is now used to re-submit tasks that were queued
for flow control reasons.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I99c28751df32017c43490a90f4904bdabe79a270
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3555
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
2020-10-22 22:43:28 +00:00
Tomasz Zawadzki
78ad672861 lib/event: remove legacy configuration from event framework
All options -c, --config and --json are valid, but have to
point to JSON configuration file.

Adjusted UT since JSON configs don't work with --wait-for-rpc.

Since this removes last reference to legacy INI configuration,
updated conf library to no longer mention the deprecation.
All uses of conf library are for explicit reason and not
related to SPDK event framework configuration.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ic9a702465982daf715ce1c2ab863c48584734611
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4752
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-10-22 17:08:41 +00:00
Tomasz Zawadzki
cda1d2cf11 build: remove unused references to conf library
- Removed slew of conf.h includes
- No longer require mk vars that include conf

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ica7e8e8bf1d4a5d0b0200bfe689aa13afd77bfaf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4746
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
5b1e154a5c bdev: remove legacy config support
This patch removes legacy config support in bdev layer.
All options through the legacy config are already reflected in JSON.

Removed configuration can be set via `bdev_set_options` and
`bdev_set_qos_limit` RPC.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I24c365625540659cad425268d2aa41e3bf279d5d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4645
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
5f90d0c016 accel/ioat: remove legacy config support
This patch removes legacy config support in ioat accel module.
All options through the legacy config are already reflected in JSON.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iebaeef4cf648874700238f6167462f4958546be2
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4635
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
ceea308887 lib/blobfs: remove legacy config support
This patch removes legacy config support in blobfs library.
Mostly reverting patch:
(97f3104) blobfs: Add conf parse for blobfs.

CacheBufferShift option in legacy config is not tested in CI,
and never received corresponding RPC.
If required the RPC can always be added later on.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I54e39f069047a243b2186ae5ea225ed452180488
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4668
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
d3e99f4891 module/virtio_blk/sci: remove legacy config support
This patch removes legacy config support in virtio_blk and virtio_scsi bdev modules.
Most of the options through the legacy config are already reflected in JSON.

Exception to that is "[VirtioPci] Enable" option, which worked for virtio_blk
and virtio_scsi. It then on initialization scanned virtio_pci devices for
presence of blk/scsi. There is no RPC that scans it on demand, nor was it used
in any of the tests. So this patch removes that functionality,
if needed this can be reintroduced as new RPC (or two for each backend).

Module initialization for bdev_virtio_scsi changed to be synchronous.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib0978a4a084795e94993bb9122bd73ed49e14b96
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4667
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
dbc7e89f58 subsystem/nvmf: remove legacy config support
This patch removes legacy config support in nvmf subsystem.
All options through the legacy config are already reflected in JSON.

Since the step in target initialization no longer does config
parsing, it was renamed appropriately.

There is no longer a need to delay this step via send msg,
since no callback is used.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I5ee080bb1f5bf954f3b00545adb16e7176c35530
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4644
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
8c66958a10 vmd: remove legacy config support
This patch removes legacy config support in vmd subsystem.
All options through the legacy config are already reflected in JSON.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I45f66809a889b39a05bf4f6856431d922b497c76
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4641
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
269efef886 bdev modules: remove legacy config support
This patch removes legacy config support in:
 crypto bdev module
- nvme bdev module
- ocf bdev module
- rbd bdev module
- pmem bdev module
- iscsi bdev module
- raid bdev module
All options through the legacy config are already reflected in JSON.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I00213365f52d3de1012493c14d4ea5fc537e595e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4673
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
b1f76f512f lib/vhost: remove legacy config support
This patch removes legacy config support in vhost_blk/scsi library.
All options through the legacy config are already reflected in JSON.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ia63651cdb7433267d1a8839a1739e68b436e5d08
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4621
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-10-20 02:42:16 +00:00
Changpeng Liu
6bd7bd0da2 vhost: deprecate internal vhost library support
The internal vhost library is used when DPDK's version
is older than 19.05 and the experiemntal vhost nvme
target.

For the CONFIG_INTERNAL_VHOST_LIB option, SPDK doesn't
enable this option by default over one year and CI
doesn't cover it either, so we may remove it with
this release.

As for the vhost-nvme target, since we are developing
a new vfio-user target solution, it's OK for us to remove
it now.

Change-Id: Ib2cce1db99cd09754307c2828b3187f2d4550304
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4562
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-20 02:42:16 +00:00
Tomasz Zawadzki
5acae9d6f7 module/uring: remove legacy config support
This patch removes legacy config support in uring bdev module.
All options through the legacy config are already reflected in JSON.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I79f9b4458af2fdf0e4683f9cd82b64e9658c5fcb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4672
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-19 09:55:43 +00:00
Tomasz Zawadzki
f0ef3b2fcb module/split: remove legacy config support
This patch removes legacy config support in split bdev module.
All options through the legacy config are already reflected in JSON.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Iaa8e546bbd6357ba93af8164e2f3d8e497fcec06
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4634
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-19 09:55:43 +00:00
Tomasz Zawadzki
421ced38b3 module/passthru: remove legacy config support
This patch removes legacy config support in passthru bdev module.
All options through the legacy config are already reflected in JSON.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I53ee6e78d210c155a31ae6204472e89756e43e65
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4633
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-19 09:55:43 +00:00
Tomasz Zawadzki
2eda3fa095 module/null: remove legacy config support
This patch removes legacy config support in null bdev module.
All options through the legacy config are already reflected in JSON.

While here, added allocation failure check for g_null_read_buf.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I59a28a402e0c61e5636da623bc109e15b36f0584
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4626
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-10-19 09:55:43 +00:00
Tomasz Zawadzki
98fde363f3 module/malloc: remove legacy config support
This patch removes legacy config support in malloc bdev module.
All options through the legacy config are already reflected in JSON.

Note that malloc_disk_count needs to be reset for each module initialization.
Otherwise for cases where bdev submodule is reinitialized, the count in
malloc names would grow. This should be addressed separately by making name
parameter required, rather than optional. Which is the case for most (all?)
of the other bdev modules.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ida93c74e7b0be7da19e11d5cc3fda9112824782f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4625
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-19 09:55:43 +00:00
Tomasz Zawadzki
ccdbe656be module/gpt: remove legacy config support
This patch removes legacy config support in GPT bdev module.

Please note that disabling probing only for GPT partition does not
have a corresponding RPC.

Similar functionality is possible for all bdev modules at once via
`bdev_set_options --disable-auto-examine` RPC then followed by
`bdev_examine` RPC for particular bdev.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I9a106b3dc0953166264b11c2e95c4433163fcd3c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4624
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-10-19 09:55:43 +00:00