Commit Graph

247 Commits

Author SHA1 Message Date
paul luse
5b03dd938c module/idxd: accel framework plug-in for idxd
Docs, RPC, unit tests, etc., will follow.  Notes:

* The current implementation will only work with VFIO.

* The current implementation supports only the existing accel
framework API. The API will be expanded for DSA exclusive features
in a subsequent patch.

* SW is required to manage flow control, to not over-run the work queues.
This is provided in the accel plug-in module. The upper layers use public
API to manage this.

* As we need to support any number of channels (we can't limit ourselves
to the number of work queues) we need to dynamically size/resize our
per channel descriptor rings based on the number of current channels. This
is done from upper layers via public API into the lib.

* As channels are created, the total number of work queue slots is divided
across the channels evenly.  Same thing when they are destroyed, remaining
channels will see the ring sizes increase. This is done from upper layers
via public API into the lib.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ifaa39935107206a2d990cec992854675e5502057
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1722
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-04-23 15:48:32 +00:00
paul luse
e58e9fbda8 lib/idxd: add low level idxd library
Module, etc., will follow. Notes:

* IDXD is an Intel silicon feature available in future Intel CPUs.
Initial development is being done on a simulator. Once HW is
available and the code fully tested the experimental label will be
lifted. Spec can be found here: https://software.intel.com/en-us/download/intel-data-streaming-accelerator-preliminary-architecture-specification

* The current implementation will only work with VFIO.

* DSA has a number of engines that can be grouped based on application
need such as type of memory being served or QoS. Engines are processing
units and are assigned to groups. Work queues are on device structures
that act as front-end groups for queueing descriptors. Full details on
what is configurable & how will come in later doc patches.

* There is a finite number of work queue slots that are divided amongst
the number of desired work queues in some fashion (ie evenly).

* SW (outside of the idxd lib) is required to manage flow control, to not
over-run the work queues.This is provided in the accel plug-in module.
The upper layers use public API to manage this.

* Work queue submissions are done with a 64 byte atomic instruction

* The design here creates a set of descriptor rings per channel that match
the size of the work queues. Then, an spdk_bit_array is used to make sure
we don't overrun a queue.  If there are not slots available, the operation
is put on a linked list to be retried later from the poller.

* As we need to support any number of channels (we can't limit ourselves
to the number of work queues) we need to dynamically size/resize our
per channel descriptor rings based on the number of current channels. This
is done from upper layers via public API into the lib.

* As channels are created, the total number of work queue slots is divided
across the channels evenly. Same thing when they are destroyed, remaining
channels with see the ring sizes increase. This is done from upper layers
via public API into the lib.

* The sim has 64 total work queue entries (WQE) that get dolled out to the
work queues (WQ) evenly.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I899bbeda3cef3db05bea4197b8757e89dddb579d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1809
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-04-23 15:48:32 +00:00
Seth Howell
5fbd7a9e55 mk: add nvme-cli make fragment to mk dir.
This has historically been maintained in our branch of nvme-cli.
However it makes updating the libraries that link to nvme-cli
prohibitively clunky from a CI perspective.

Moving this fragment here brings us more into line with how we handle
other projects we link to like rocksdb.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ia31282983a6f6b0d9569b6a97c38b5f3c9423492
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1932
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-04-22 09:22:44 +00:00
Seth Howell
8671edd925 make/unittest: don't link against shared env lib.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I65b916c9d6bc34bd0370894d88ec8d9c0801c949
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1814
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2020-04-22 09:21:55 +00:00
Seth Howell
96250f5b3d make: add a map file for libspdk_accel.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I49f0011d0cdb22ae8830d71b3ba554f85f83892e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1400
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-04-03 06:31:46 +00:00
Seth Howell
91003e5287 make: don't link vpp into fio plugins.
Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I96e29cd0681d767d22e31973950655b974ce9031
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1397
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-03-26 08:12:04 +00:00
Ziye Yang
465f4c1254 sock/uring: Add the async network I/O support for socket
This patch is used to add the async network I/O support in
sock layer. If code is configured with I/O uring, --with-uring,
we can use io uring in Linux (version >=5.4-rc3).

PS: We also make VPP's default priority > uring, because
for the iSCSI or sock test linked with VPP. It tests VPP with
a given address (which is not a special VPP can only open address),
so using uring can also listen those address succefully. And if we make
uring with priority > VPP, actually, VPP will not tested in those cases.

Additionally, the current CI pool is not ready for test, we need
wait for the CI system ready. And I test on my local platform, it works.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: Ie8ee3c8ddf8d2a7264f2b382376733e002816dcc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/952
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-03-20 08:41:28 +00:00
Seth Howell
193927830d make: rev SO versions individually for libraries.
This will allow us to keep track of compatibility issues on a
per-library basis.

Change-Id: Ib0c796adb1efe1570212a503ed660bef6f142b6e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1067
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-03-18 08:02:30 +00:00
Ben Walker
eb98488946 sock/posix: Internally buffer reads.
Do large reads from the socket and buffer into a pipe.

Change-Id: I0a1bc5b356cb8c403048ab3b22a159332f085e6b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/447
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-03-17 08:23:07 +00:00
Seth Howell
c5911f0224 mk: bump the shared object major version to 2.
This is to indicate the ABI breakage in the bdev library. A function's
argument list was changed which breaks both backwards and forwards
compatibility.

Going forward, all backwards compatibility breaking changes should be
marked with a rev of the SO major version for that library. All forwards
compatibility breaking changes should be marked with a rev of the SO
minor version.

Change-Id: I35e45c102c5c6de3c684919a10e5116f8f2c375f
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1066
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-03-04 10:04:52 +00:00
Seth Howell
07b909b08c Revert "shared_lib: bump version of all .SO to 1.1"
This reverts commit 2ce1406a59.

It seems that the typical practice for making SO version changes is
this. When a change has been made to an ABI that breaks backwards
compatibility, the major version should be updated. When a change has
been made which breaks forward compatibility (i.e. applications linked
to this ABI are not necessarily compatible with older versions of the
ABI due to added symbols).

We should update the major ABI version now, and
then going forward follow the pattern mentioned above.

Change-Id: I0ecd4ae64398dc1a6d2ce505303d012397a2047b
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1065
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-03-04 10:04:52 +00:00
Tomasz Zawadzki
2ce1406a59 shared_lib: bump version of all .SO to 1.1
The original SOVERSION was not updated since its introduction,
two years ago.

This will serve as a cut-off point for any changes done before.
Going forward each SO will be versioned separately whenever
ABI breaking change occurs.

Version 1.1 will point to changes done up to
release of SPDK 20.01.
By SPDK 20.04 some of the libraries might update to 1.2.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I9ef6c1d345a66b7683efa61e28edb5c2ba9b7c5a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1009
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-02-27 10:14:32 +00:00
Maciej Szczepaniak
35ce722145 make: Compile with libpmem if --with-pmdk is set.
Signed-off-by: Maciej Szczepaniak <maciej.szczepaniak@intel.com>
Change-Id: I120c4a1e04df5cf79900b6b84d7392f57ca25238
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/648
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
2020-02-25 10:47:15 +00:00
paul luse
e753402aa1 lib/accel: add RPC to select which accel module should be used
Startup RPC only to avoid issues trying to switch while in use.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I7f1801905de0927f3610c4065182dd8fa88cdfdd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/660
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2020-02-18 08:05:34 +00:00
paul luse
aa7a13afc7 global: rename copy to accel
The copy engine library, modules and public APIs have been renamed.
Use of the word `copy` has been replaced with the word `accel`
short for accelerator in preparation for adding new capabilities
in the future. Additionally, APIs for what was previously called
the `memcpy` engine have been renamed to identify the engine as a
software accelerator.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ia607aa718416146fbba1e6792b8de0f66bd8a5de
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/576
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2020-02-18 08:05:34 +00:00
Evgeniy Kochetov
e5bf4f061a make: Fix shared lib build
Build with shared libraries fails if there is '.a' anywhere in the
path to spdk, e.g. '/home/user/dir.auto/spdk'.
'subst' function in spdk.lib.mk is replaced with substitution
reference, that alters only suffixes.

Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Change-Id: I5dd06ab95b85d5dd93becff001cef8e78daa0b81
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/669
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2020-02-11 08:37:16 +00:00
Wojciech Malikowski
e2e6254651 bdev/ftl: Remove NVMe dependencies
This patch changes FTL bdev to vritual bdev.

Change-Id: I7b96af56053874b670a76b910a846837396119d9
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479703
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2020-01-15 12:24:44 +00:00
Wojciech Malikowski
843f296e2e lib/ftl: Replace Open Channel dependencies with zone bdev API
This patch replaces NVMe Open Channel API usage
inside FTL library with corresponding zone bdev
API calls. This include following calls:
 - spdk_nvme_ctrlr_cmd_get_log_page -> spdk_bdev_get_zone_info
 - spdk_nvme_ocssd_ns_cmd_vector_reset -> spdk_bdev_zone_management
 - spdk_nvme_ns_cmd_read -> spdk_bdev_read_blocks
 - spdk_nvme_ns_cmd_write_with_md -> spdk_bdev_write_blocks

Change-Id: I1b5a6863d9ce72f4af1cfbb0e449fc1a5b638144
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/479702
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2020-01-15 12:24:44 +00:00
Seth Howell
de467ec91d env_dpdk/rpc: add rpc to get memory stats.
This will enable us to create a helper applciation to print memory
statistics for a given SPDK application.

Change-Id: I0ad5a8163166a5d67d9a5533303ec0858297ba6b
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/477511
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-12-13 11:05:57 +00:00
Mateusz Kozlowski
cabbe1b179 bdev/zone: Register/unregister zoned bdev
Add registration and unregistration of block zoned bdev. Attach it to the
underlying bdev during creation and unattach at deletion.

Signed-off-by: Mateusz Kozlowski <mateusz.kozlowski@intel.com>
Change-Id: I773aff6c7609952f28c02dd1794f0529a781b2e1
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468033
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-11-15 20:27:14 +00:00
Mateusz Kozlowski
195be29e9b bdev/zone: Rpc functions for zoned block bdev
Added rpc handling for creation and deletion of zoned block bdev

Signed-off-by: Mateusz Kozlowski <mateusz.kozlowski@intel.com>
Change-Id: Ib7b6f3928511aa840cbe098472682c951e92e5ed
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468031
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-11-15 20:27:14 +00:00
Mateusz Kozlowski
76fb903a74 bdev/zone: Create and delete zoned block device vdev
Adding new bdev module - zoned block device virtual bdev. It should be
possible to build on top of a regular (ie. non-OCSSD) bdev and surface
a zoned device API instead.

Added create and deletion functions for this bdev.

Signed-off-by: Mateusz Kozlowski <mateusz.kozlowski@intel.com>
Change-Id: Ia383483007117d1c826298fd391467a51fa2fe4e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/468030
Community-CI: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-11-15 20:27:14 +00:00
Alexey Marchuk
2ad7573393 build/rpc: Add git sha1 to SPDK version macro
Retrieve git sha1 during SPKD build, add sha1 to SPDK_VERSION_STRING
macro and to spdk_get_version RPC method
This might be useful for issues reporting/investigation

Change-Id: I2bfed752839036bf914b546066ed4d23ef4c99b8
Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/474004
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-11-14 03:46:17 +00:00
Konrad Sztyber
af4e2a3277 lib/vmd: hotplug / hotremove support
This patch adds support for hotplug / hotremove detection for devices
behind the VMD.  The detection acts similarly to the one implemented for
regular PCIe devices, that is user has to periodically call probe
function.  Additionally, for applications not using SPDK's event
framework, spdk_vmd_hotplug_monitor has to be called periodically as
well.

Change-Id: I9f6839560efcf16c839b01976639d835f119cb47
Signed-off-by: orden smith <orden.e.smith@intel.com>
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/472741
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-11-06 14:47:39 +00:00
Ben Walker
913f780e10 test: Add mocks for sendmsg and recvmsg
These will be used from the posix sock layer.

Change-Id: I6427d9e9aee889e11ba7e36876112a5aba449e31
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/471751
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-11-01 17:56:16 +00:00
Tomasz Kulasek
805d994281 lib/nvme: add NVMe character device
NVMe character device implementation. This patch adds implementation
of IO producer using CUSE library. It allows to create nvme device
nodes in linux kernel for controller as well as for namespace and
process ioctl requests as usual from linux environment.

Both devices (controller and namespaces) are exposed as character
devices.

To compile NVMe CUSE module use "./configure --with-nvme-cuse".

Names for created CUSE devices can be retrieved using
spdk_nvme_cuse_get_ctrlr_name() and spdk_nvme_cuse_get_ns_name().


Signed-off-by: Jim Harris <james.r.harris@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: I0fc9a9a1ef3c9c2b3112d07c2b4b1f8d49665ee1
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466917
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-10-24 23:43:59 +00:00
Xiaodong Liu
3ce759a1d5 blobfs: add blobfs_detect RPC
Add module blobfs_bdev as a general module to simplify
the operations of blobfs on bdev. Then its functions
can be utilized by other libraries or apps.

blobfs_detect can be used to detect whether there is
one blobfs on given bdev.

Change-Id: Ib425574816061dc945fb652b539f791a44097a43
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466486
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-10-09 05:02:00 +00:00
Ben Walker
7171d7d4e3 sock/posix: Call set_recvbuf internally
Instead of having two places in the code that do the
syscalls to set up the recvbuf size, just call the
function we already have.

Change-Id: I7d098fc7a39d0593c58fbe05b1f0b14f8b6a360d
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470509
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-10-08 15:48:29 +00:00
Ben Walker
8efa583f13 test: Add a mock for writev
This is useful for testing partial sends in the socket layer.

Change-Id: Ib5bb62d641ad8d6a23a662261f7c72f5ba55c813
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470520
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-10-08 15:48:29 +00:00
Jim Harris
c2b3c18b5c mk/spdk.common.mk: remove -std=c++0x
This isn't really needed, and actually hinders
things like Seastar which use more recent C++
variants.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7ce3d0578d6832a5d2b26c661bf7a7e2119bcac5

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/470079
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-10-03 23:02:05 +00:00
Seth Howell
a19e674be0 make: always fail if mk/config.mk doesn't exist.
We previously were failing for all make targets except for clean if the
mk/config.mk file didn't exist, but we should also be failing to
perform make clean if we don't have this file. It turns out you end up
erroring out in the next line anyways. Also, if you make that include
optional, make still fails because we don't have any kind of environment
specified and spdk.common.mk depends on that variable being defined.

There may be some benefit to being able to run make clean if we don't
have a valid configuration file, but right now, there are a lot of
things that depend on the configuration file, such as which environment
file to make or clean and which directories need cleaned.

Change-Id: I2a9c1e982c01ba76b67b153ee7336402300b76b0
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466346
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-09-19 05:03:49 +00:00
Seth Howell
a988124db9 make: override scan-build for unit tests.
The unit test mock macro function doesn't mesh well with the scan-build
static analysis tool. So we need to reset cc to the user provided value
or default before running the unit tests.

Change-Id: Ib636f6e396b09f9bf1cb8f930e6befa56ce99848
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466816
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-09-11 16:30:38 +00:00
Ben Walker
52ccf4b761 sock/posix: Increase SO_RECVBUF size by default
Everywhere we use this, we increase the size to 2MB.
Just make that the default behavior.

Change-Id: Ie174419a09df1792a0c7311eddd0c2dcefa267b5
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466991
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-09-10 17:48:49 +00:00
Seth Howell
28099e0ed9 make: add dependencies for the rte_vhost build.
The dependencies between vhost and rte_vhost were not added during
earlier changes. This change moves the rte_vhost directory up to the
level of the other libraries and adds the proper dependencies for when
it is linked.

Change-Id: I089de1cd945062b64975a0011887700c0e38bb0f
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/467700
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-09-09 22:15:24 +00:00
Seth Howell
5c62618f26 module/event: add subsystem interdependencies
The SPDK application framework defines a list of event subsystem
dependencies. When linking against individual shared libraries, it is
useful for this dependency structure to be codified in the shared
libraries themselves.

For example, when linking a bdev based application against
libspdk_bdev.so, one might wish to only specify this shared object at
link time. However, when you actually run the application, it will fail
to start because it is not linked to the copy and vmd subsystems.
However, once thesedependencies are added, one can effectively link
against only the exact subsystems they need and any dependent subsystems
will be linked in automatically.

Change-Id: Ic986281a162ac20b523486e9f8cccf4a0787afd7
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466081
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-08-29 20:16:18 +00:00
Seth Howell
407e88fd2a lib/mk: update OCF build.
The OCF build was broken by some of the recent changes
to the Makefiles. This change aims to fix that by separating out the ocf
environment from the ocf bdev.

Change-Id: Id445340033898e9ae70a4bcfc799951110762d55
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/465808
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-08-27 18:49:56 +00:00
Seth Howell
7e1881cef9 module: add shared lib deps for all modules.
This will allow us to link individually against dpdk module libraries
without having to define those libraries dependencies.

Change-Id: Ief2140ec0fadd970aba990dab333d603dfb46317
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/465704
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-08-27 18:49:56 +00:00
Seth Howell
f66caa1f80 make: start adding inter-lib dependencies.
This will make it much easier to link against some subset of spdk
libraries when building SPDK applications.

Change-Id: I8ad95a001965288a8b5e38eb252391fef68d7138
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/465194
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-08-27 18:49:56 +00:00
Seth Howell
9d14641dd5 mk: add a lib deps file.
By adding this file and modifying the way we define a couple of
variables in the makefiles, we can actually avoid having to redefine
the library dependencies on a per file basis.

Change-Id: Ieab4aa1021b0341fc21e3b65677a9ad7f70559c2
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466063
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
2019-08-23 17:48:35 +00:00
Seth Howell
07fe6a43d6 mk: move the bdev modules under module directory.
This is more accurate to what they are, and will make defining library
dependencies much simpler. This change in directory does not affect the
final placement of naming of libraries at the end of time.

Change-Id: Ic48a9233dff564e39ce357a9ea0a111ea2b6414b
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/465454
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Broadcom SPDK FC-NVMe CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-08-22 16:29:49 +00:00
Seth Howell
2a3b6b4179 mk/lib: Build subdirs before lib dir
The subdirectories of libraries are currently listed as dependencies of
the library. If you look at the actual symbols they contain, the
opposite is true. For all of the bdevs, event subsystems and other
library directories, the subdirectories actually rely on symbols
contained in the parent directory.
If we want to create the proper inter-library dependencies in SPDK at
compile/link time, then we need to respect the order of these symbol
dependencies when executing make (e.g. make libspdk_event.so before
libspdk_event_bdev.so). Otherwise we create a loop of dependencies in
the make system.

Change-Id: Idcbb050b2709f575d0cec46b481824d6313612b4
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/465191
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-08-15 20:32:28 +00:00
Seth Howell
e29c6fe133 mk: force --no-as-needed for spdk libs.
When building shared libs, we specifically don't set the --as-needed
flag so that we still link against libraries that have constructor
functions. LD reports on its man page that the default behavior is
equivalent to --no-as-needed.

For RHEL based distros like fedora and centos this works fine. While
the LD man page shipped with ubuntu distributions also states that
--no-as-needed is the default, they don't respect that behavior and do
--as-needed linking unless you force them with --no-as-needed.

Change-Id: I914ab849323de198af5c5e53fffb1f57fcaff5fe
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464621
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-08-12 21:12:16 +00:00
Darek Stojaczyk
35a4f3a4bd mk: clean up BLOCKDEV_MODULES_LIST in spdk.modules.mk
Reorder BLOCKDEV_MODULES_LIST entries - put dependant
libraries first. Also break lines after 80 characters.

Change-Id: Ib09b28938253435f335c4e1ebe83928ab551194c
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459909
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-08-08 22:41:30 +00:00
Ben Walker
73439e6f46 build: Detect compiler toolchain in configure
We need to know this up front.

Change-Id: I3a9ceb90cf62eacbf3fdf518a9ccb4c4978b3a05
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/463014
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-26 16:22:27 +00:00
Ben Walker
0e6ddfd66a build: Allow the user to set target architecture in configure
Change-Id: I7f2a1464c0c8872e5e7856a830aaf7ea8cdf4e81
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/463013
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-07-24 22:55:06 +00:00
Ben Walker
ebd32197ba build: Allow TARGET_ARCHITECTURE to be specified on ARM
Previously it always forced armv8-a+crc. Now it only does that
if not otherwise specified.

Change-Id: I7c1d73fca8e89779cc3e8780a38f52975488d97a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/463012
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-07-24 22:55:06 +00:00
Ben Walker
7dbdf87f63 make: Allow users to specify the target build architecture
make TARGET_ARCHICTECTURE=broadwell

Change-Id: I868c2f5c203dfc1a4e6cf48c39b391e174d5d33b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/461601
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-07-23 17:01:17 +00:00
Tomasz Kulasek
14a62d67ac net/vpp: allow to compile with vpp
This patch prepares environment to allow compile applications and
libraries with VPP.

Change-Id: Icc067c17b57932062afa5e6a67791bd58de3a2cc
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456461
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2019-06-27 08:23:08 +00:00
Wojciech Malikowski
9d92b9d420 examples/nvme/perf: Optional VMD enumeration
New flag -V was added for VMD enabling.

Change-Id: Ie0feae1bdfce39ad246f57411d00587c6417d21d
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450926
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-18 03:59:36 +00:00
Seth Howell
3fb4a66d7d bdev: Create a new delay vbdev module
This module simply sits on another virtual bdev
and adds a simulated average and p99 latency to that drive.

Change-Id: Ie9fc91e27585fd0636cb7dc845cb41744bf24625
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453594
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-31 14:37:26 +00:00