8476 Commits

Author SHA1 Message Date
Vitaliy Mysak
c3d044cbd7 ocf: use vbdev_ocf_mngt_ interface in register path
This patch is a preparation for adopting ocf_cache_trylock() function.
Register OCF bdev path uses vbdev_ocf_mngt_ interface now
Register stays synchronous blocking operation in this patch,
 but with adoption of ocf_cache_trylock() function,
 register will be asynchronous, in which case we
 want to use mngt_ interface.

This series of OCF patches will enable WriteBack support for OCF bdev.
There is a lot of preparation before we will be able
 to actually implement WriteBack.
Dependencies look like this:
- WriteBack
  - Cleaner
    - trylock
  - Persistent metadata
    - asynchronous management API

Cleaner is a background agent that does synchronization
 of data between cache and its cores.
 Cleaner usually runs every ~30 seconds to perform cleaning.
 The synchronization is a simmilar operation to OCF management flushes.
We need cleaner for WriteBack because only WriteBack mode
 produces dirty data that cleaner needs to deal with.
Cleaner requires adopting trylock() because in current version
 cleaner uses management lock when performs cleaning,
 which may lead to deadlocks if cleaner runs on
 the same thread as management operations.

Peristent metadata is functionality of OCF
 that allows to restore cache state after shutdown
We need persistent metadata in context of shutdown
 with some data being dirty which may only happen
 when using WriteBack mode
Support for persistent metadata requieres
 asynchronous OCF API because in current version
 we will have a deadlock during metadata initialization
 because it is required to be done on different thread
 than cache initialization.

Change-Id: I45d84a5fa0c96581d522050dad186b06d489226e
Signed-off-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455225
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-22 17:57:31 +00:00
Konrad Sztyber
bdbd32b4c7 lib/ftl: minor ftl_rwb_fill cleanup
Change-Id: I69ee623d9e5646b7a793313dac0fb82ef04035f3
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453994
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-22 16:17:23 +00:00
Konrad Sztyber
36ff0eeb88 lib/ftl: mirror writes to non-volatile cache
Apart from writing the data to OCSSD, mirror the latest two bands of
data on the persistent write buffer cache. Currently the data is only
sent there, further patches will add metadata support, shutdown recovery
and L2P updates.

Change-Id: Ief05d0c23fa0e25bd6085e0ce3e1528d6736d174
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450266
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-22 16:17:23 +00:00
Konrad Sztyber
aba66c0644 lib/ftl: added ftl_io_shrink_iovec
This function can be used to reduce the number of iovecs on exisiting IO
structure. It's useful when only the maximum number of iovecs is known
at the ftl_io allocation time.

Change-Id: Ief06634446c65889401f5e0b07b2d1ce7912af34
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450630
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
2019-05-22 16:17:23 +00:00
Konrad Sztyber
c139a7d9ba lib/ftl: persistent cache initialization
Change-Id: I608d420bd115a70d60ee6995220dbfafb7a00731
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449242
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-22 16:17:23 +00:00
Tianyu yang
a8b8badb3c bdev/aio: An error code can be returned via the rpc interface.
When we create AIO using the RPC interface, we can use the error
message to return to determine what exceptions have occurred.

Change-Id: I537f1a7d1053dcc27960de004fd0b4aea49919e9
Signed-off-by: Tianyu yang <yangtianyu2@huawei.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452313
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-22 16:12:40 +00:00
Tomasz Zawadzki
3c45798173 test/bdevio: enable RPC listening socket
Previously it was explcitly set to NULL, so that
the application didn't even setup up the socket.

Change-Id: I2174fe0ff5790efd6578807f17702978cd9cf451
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455197
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-22 15:51:36 +00:00
Tomasz Zawadzki
3f1cda12e4 test/bdevio: enable every kind of RPC to the app
Adding app_rpc and bdev_rpc allowing every kind of RPC
interaction with the app.

Change-Id: I738095686dc8ad61101a998b473df63cddc490a2
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455196
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-22 15:51:36 +00:00
yidong0635
2dac00b9a4 vm_setup: fix pop-up dialogs in auto installing package
Installing this package "open-isns-utils" always with pop-up dialogs,
we don't want to these dialog, so add --no-install-suggests
--no-install-recommends to prevent it.

Use sudo apt-get remove --purge * to simulate.

Change-Id: I0b892b9e0c88c82ab9461a92e71dc0d9823ecaf9
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455333
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-22 15:37:39 +00:00
paul luse
3d6305b9fd bdev/compress: fix issues with multi-thread
Submission logic was incorrect and completion logic was not
present yet.  This passes now with multi-thread bdevio test.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ia8ed1c123be511240d93503a2c5e501ccad445bf
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455018
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 15:36:48 +00:00
paul luse
4397e38782 bdev/compress: add UT for the poller
Adding these tests identified a small fix in the code under test
that is also included here.  The compressdev 'produced' field
is unsigned and reduce is expecting a negative errno in the
callback.

Change-Id: I28ab11ee3ef54768a9d6ccd26282cf7dd022be43
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454806
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 15:36:48 +00:00
paul luse
3cf1cd0b14 bdev/compress: use comp bdev assigned queue pair in poller
Previously hardcoded to 0 for iniital dev/test.

Change-Id: If3b98b0083ff52366eb4deaadb5ffb6a277f904c
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454681
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 15:36:48 +00:00
paul luse
96fe0fb199 bdev/compress: updates to the poller function
Two quick updates, support dequeueing more than 1 IO at a time which
is now more likely given that we can queue up compression operations
(others can come in qhile one is on the queue).

Dropped the max inflight value as it was in there as a palceholder
and now it makes sense to give it a sane value. Also updated the value
passed to reduce to signal an error.

UT for the poller will come in the next patch...

Change-Id: I14d6aeb98aa1e193c498e1549e37eca0e4c56c31
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454680
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 15:36:48 +00:00
Jim Harris
37184dd471 nvme: add nvme_free_request() variant that takes qpair
This avoids dereferencing the request to get the qpair
in cases where we already know the qpair.  Adding a new
variant instead of just modifying nvme_free_request()
since there are 72 calls to this function and I don't
want to change all of them.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ifd6fd964e546bcd71ff180fd71d5bf5cbab79d4f

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455287
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 14:51:01 +00:00
Jim Harris
ef1f844395 nvme: add qpair parameter to nvme_complete_request
In some cases we have the qpair already when calling
this function.  So pass the qpair to avoid having
to get it from the request.  This shows about a 3%
performance improvement for high IOPs single core
tests.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I22fcca560492f4e7cf5ffedd252e41a027d0dd79

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455286
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 14:51:01 +00:00
Jim Harris
a9b174e71e build: fix duplicate spdk.common.mk includes
mk/nvme.libtest.mk includes mk/spdk.common.mk, but all
of the Makefiles including mk/nvme.libtest.mk were
also including mk/spdk.common.mk unnecessarily.  So
remove the spdk.common.mk include from all of the
offending Makefiles.

This was relatively harmless, although it would cause
weird things like CFLAGS and LDFLAGS getting duplicated
when building.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie60637db3c19a2ead097562b2adf6573dbe27472

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455321
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 14:51:01 +00:00
Jim Harris
9018dd60c6 examples: rename ioat perf tool to ioat_perf
This avoids conflict with the nvme perf tool.  PGO
gets confused during building - we may have data for
nvme/perf which it then tries to use when building
ioat/perf.  Renaming the ioat perf tool fixes that
problem.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib1084d56d671e44027ea05f453075a723f067580

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455320
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 14:51:01 +00:00
Jim Harris
bf647c168a nvmf: increase default max num qps to 128
This matches the Linux kernel target.  Users can
still decrease this default when creating the
transport (i.e. -p option for nvmf_create_transport
in rpc.py).

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Icad59350a2cd35cfc4ad76d06399345191680c05

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454820
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 14:50:05 +00:00
Jim Harris
d96abc9f0d test/nvmf: remove -p num_queues restrictions
There's a bug in Linux 5.1 (possibly 5.0 too) where
the kernel initiator driver crashes if it can't get
one queue per CPU.  This will get fixed eventually,
but for now we need to remove the cases where we
restrict the number of queues per controller so that
we can test on newer kernels.

Even on cases where we're testing the SPDK initiator,
there's no real need to restrict the number of queue
pairs.

The kernel will eventually get fixed, but we should
be testing with default behavior anyways (the kernel
wants lots of queues).  We'll also want to add some
regression tests to make sure the kernel doesn't
break again.  But that will all come later.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I9979e6d94456e075688b822b042936b63e518a4a

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454819
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 14:50:05 +00:00
Darek Stojaczyk
d2e2758e0d test/qos: precalculate bandwidth limits
Don't do it at runtime, do it just once on script initialization.

Change-Id: Idb345fc0f72d3a41072d830b11a520584ec8b321
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451886
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-05-22 10:21:49 +00:00
Darek Stojaczyk
ac8921f56c test/qos: use a single RPC for getting bandwidth and iops stats
Do a single RPC instead of two, cache the result,
parse it twice.

Change-Id: Ib0ae19da5cb6b6db7bf1a46c3960104d1975afdd
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451885
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 10:21:49 +00:00
Changpeng Liu
c55c85f807 bdev: complete parent IO only when all the children IO are finished
When parent IO was splitted into several children requests, SPDK
may return parent completion callback with error status before
all the children requests are finished.

Change-Id: I63221a0ae1a5925a7fcd9744b4f5d8079c641252
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453611
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-22 08:18:40 +00:00
Darek Stojaczyk
75a6265de9 rpc: fix segfault on get_spdk_version request with extra params
We tried to send two responses for a single RPC request.

Fixes #798

Change-Id: I37eebd6a1212f6ab2c026b7587415e42c3c7417e
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455220
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-22 04:36:47 +00:00
Ben Walker
e036215fe9 thread: Add a mechanism to exit a lightweight thread
Lightweight threads may now be exited by calling
spdk_thread_exit() within the thread. The framework
polling the thread can release the resources associated
with that lightweight thread by calling
spdk_thread_destroy().

Change-Id: I6586b9d22556b3874fb113ce5402c6b1f371786e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455319
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-22 04:23:17 +00:00
Ben Walker
f658a867f9 thread: Remove unknown_tsc tracking
This was actually always 0.

Change-Id: I33a14a9f213e8c7159f00f57c88f12cfdf1d3d60
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455318
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-22 04:23:17 +00:00
Darek Stojaczyk
3a0627f069 vhost: remove 2MB memory region size restriction
We no longer have any assumptions about vhost memory regions
size being a 2MB multiple, so we can get rid of the security
check preventing some vhost sessions from being initialized.

It will be necessary for virtio-vhost-user, whose memory comes
from PCI BARs and its size may not be a 2MB multiple.

Change-Id: I48f9bc20f4c61aefdddf39ade875867148f0ed75
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454879
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-05-22 00:53:08 +00:00
Darek Stojaczyk
ac498fa31c vhost: use DPDK APIs to split non-contiguous virtual memory buffers
Currently, we translate each 2MB chunk to manually check
if it's contiguous with the previous one, but there are
rte_vhost APIs that do it way more efficiently.

rte_vhost_va_from_guest_pa() was introduced in DPDK 18.02,
but was backported to 17.11 as well, so we don't even need
any RTE_VERSION ifdefs to use it now. This function
calculates the remaining region size instead of trying to
translate subsequent 2MB chunks over and over.

The previous rte_vhost_gpa_to_vva() was deprecated a long
time ago and after this patch we no longer make any use of
it.

DPDK usages of this new function check if the translated
memory region has 0 length, which seems very silly, but
let's just do it in SPDK as well.

Change-Id: Ifae8daa5f810b5a2ba1524958ad2399af700b532
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454878
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-05-22 00:53:08 +00:00
Seth Howell
5931284f74 tet/nvmf_vhost: change the link to vhost/common.sh
Rather than revert this patch, just modify the path to the common script
to enable it to pass in the test pool.

Change-Id: I33f5d89e1e118df4546f45237b55173ad30fad24
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455277
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-21 19:17:31 +00:00
Seth Howell
0372f2b3ae create nvmf+vhost test.
This will allow us to verify functionality between the nvmf target and
the vhost target and test extra code paths within the nvmf target
including multi-sgl stuff.

Change-Id: I3f5e9351c11ab896b75cd7bba7a69d95c1d031be
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451993
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-21 10:01:50 +00:00
Seth Howell
5904abae79 vhost_ut: Add tests for spdk_vhost_vq_avail_ring_get
Change-Id: Iae7d5feef98eee59560042b64921acefafeb8da7
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453439
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-21 05:29:44 +00:00
Darek Stojaczyk
ca0cdbf269 bdev: increase QoS granularity
Fixes #791

Change-Id: Iaba31bdbbff6fa7996ca3e89ac4c9e3658a2cc94
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454669
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-21 05:09:41 +00:00
Karol Latecki
6be900dae8 test/vhost: remove sourcing from integrity_vm.sh
We do not need these lines in this script as it is meant
to run as standalone in a spawned VM.

Change-Id: I172227b6a7c6271d082f21677f13f2ce810fcfcf
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455092
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Pawel Kaminski <pawelx.kaminski@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-20 20:02:27 +00:00
paul luse
45e17cb242 bdev/raid: drop max raids to 2
For UT there's no additional value in having more than 2 raid
devices, the value in having > 1 is in confirming the array
structures via create multiple and verify expected raids were
created.

Change-Id: I874b76644ccfc7d9dd307fbb179b35673b7bacc9
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454508
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 17:02:35 +00:00
paul luse
85bedfac7a test: run bdev_raid.sh nightly instead of per patch
Existing raid UT already checks the logic in the raid
unmap code to intelligently send IOs to member disks.

The additional value of bdev_raid.sh is in doing a real
data validation on writes as well as confirmation that
unmaps really result in 0's following the action. Both of
these seem worth keeping however with the UT in place to
check the logic they are a better fit for nightly testing.

Change-Id: I1a758f4681d77dbddd4185162b0dd1f46525cabd
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454668
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 17:02:32 +00:00
paul luse
02064419e2 bdev/passthru: add error check in get buffer read callback
Was simply missing.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Ia8ff3825e5d78c8814c3c57b779608ebe6be34ac
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454467
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 17:02:27 +00:00
paul luse
634af37b46 bdev/compress: use the queue pair for the current device
Had 0 hardcoded early on, this just uses the queue pair that
was pre-assigned to the compression bdev to use when interacting
with the cryptodev API.

Multi-threaded testing is coming later.

Change-Id: Iab8ea4e59169980933e9c81715baf4d41803b572
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453921
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 16:59:43 +00:00
paul luse
9790bba16d bdev/compress: add UT for vbdev_init_compress_drivers()
Change-Id: I714aad7f82050b5ca480d2c51cffaaee8f059495
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452739
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 16:59:43 +00:00
paul luse
eb71364bce bdev/compress: remove unused test function, test_error_paths()
I'm testing the error paths within each function under test, this
was here simply because I copy-n-pasted much of the UT framework
from crypto to get it jump started.

Change-Id: Id7705bcaec87e93b589360e520b330f024d3293d
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452723
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 16:59:43 +00:00
paul luse
22760306a2 bdev/compress: add UT for vbdev_compress_submit_request
Note that much of the vbdev module is still WIP including this
function (ex there are rc checks at the end when nobody yet sets
an rc) but I want to catch up on UT before adding more new code,
or at least get close.

Change-Id: I4617c215fed9fe35a68dcc5e7ebc93e48588cf5b
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452715
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 16:59:43 +00:00
paul luse
3671369a1f bdev/compress: Add UT coverage for _compress_operation()
Aas well as general test setup required for this one, additional
mocks, etc.

Had to add some asserts to make scan-build happy although I don't
see how these changes stimulated those failures.

Change-Id: Ief08f9b71ee7a836f6026d26517f5faa5f9d51ce
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451688
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-20 16:59:43 +00:00
paul luse
70165cedeb bdev/compress: queue up compression operations if we can't submit
The compressdev enqueue call will return the number of ops that
it was able to accept.  By design this will always be 1 for our
implementation so check that to make sure.

If it is 0 then we queue the compression operation up and try
again after something is dequeued in the poller.

We will also queue if we can't get an mbuf or op from our pools.

Change-Id: I7285749b4a599d1ee265e3c3e16073f25c4d7469
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451686
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 16:59:43 +00:00
Darek Stojaczyk
122cc72cd9 Revert "scsi: Simplify removing LUN when no connection is for it"
The new, simplified scsi lun hotremove path doesn't call
lun->hotremove_cb if there's no io_channel allocated for
that lun. Vhost still depends on that callback and currently,
when the underlying bdev is removed, vhost is left completely
unnotified. It keeps a dangling pointer to a scsi lun and
will eventually crash. The vhost scsi controller also can't
be removed in this case.

This reverts commit 19182431c86aa904e40c411a9897898b3e8224f0.

Change-Id: I330330fdd7d6941db070d972192481f535f62977
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454836
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-05-20 16:56:06 +00:00
yidong0635
26683005dd lib/ftl: fix scanbuild error about divided by zero.
We should add assert here to avoid be divided by zero,
that resolve this scanbuild error.

Change-Id: I12fa11cdbf0f608995f91759140408303e243357
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454539
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-20 16:55:47 +00:00
Konrad Sztyber
275f0c341c blobfs: removed unaligned memory access
Replaced (uint64_t *) dereference of (char *) variable with memcpy to
avoid unaligned memory access. It fixes the following error reported by
ubsan:

spdk/lib/blobfs/blobfs.c:1492:2: runtime error: load of misaligned
address 0x00000047a136 for type 'uint64_t', which requires 8 byte
alignment

Change-Id: I5f3f28a52b8cbe3eb30394ae7ebd0e64b683db57
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454886
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-05-20 16:55:42 +00:00
Konrad Sztyber
2fac8bc76b lib/bdev: merge buffer allocation in get_buf/put_buf
These two paths do the same thing, only spdk_bdev_io_get_buf tries to
allocate the buffer directly, while spdk_bdev_io_put_buf reuses the
returned buffer for one of the waiting requests.

Change-Id: I341a011f3a16a99de399c8b56fa73ccd9c7fe4fa
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451466
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2019-05-20 16:54:13 +00:00
lorneli
598ba408a2 bdev: add test case for splitting iovs that exceeds child_iov
Test multi vector command that needs to be split by strip and then
needs to be split further due to the capacity of child iovs.

Add a case that was not tested before. In this case, the length of the
rest of iovec array with an I/O boundary is the multiple of block size.
Expect the rest of iovec array to be submitted in the completion of
previous iovec array.

Change-Id: I5b95b1f1884a73b31709b2fd9187a8a9e9b2cd0b
Signed-off-by: lorneli <lorneli@163.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452414
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-20 09:37:59 +00:00
Konrad Sztyber
e382b437b9 lib/ftl: check config after geometry is retrieved
Moved device configuration checks inside ftl_check_init_opts to allow
for validation against the geometry of the drive.

Change-Id: I8e813d30ef5feba641517872ce068824442cfdbc
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451516
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Claire Jihyun In <claire.in@circuitblvd.com>
Reviewed-by: Young Tack Jin <youngtack.jin@circuitblvd.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
2019-05-20 09:12:08 +00:00
Jim Harris
46fef66536 test/nvmf: improve nested lvol test
Nested lvolstores take a very long time to create,
since the initial unmap will get broken into
one unmap per cluster for the underlying lvol.

This test is working fine in the test pool, but they
have much smaller SSDs.  My system has a 3.7TB SSD,
and this test times out.

So don't bother clearing the nested lvol store - this
saves unmap operations.  We still have unmap operations
when deleting the nested lvol, so increase the cluster
size for the main lvolstore to 1GiB to reduce the total
number of unmap operations.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I29a254fff443e963cd620b55e78092d6a96f8ddd

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453010
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-05-20 09:03:39 +00:00
Pawel Kaminski
357ccdd8a4 test/vhost: Fix. Remove recursive common.sh sourcing
Change-Id: I21c924f84af55571dc07342fd32b8f3805745c4f
Signed-off-by: Pawel Kaminski <pawelx.kaminski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454880
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-20 09:02:33 +00:00
Jim Harris
86d8214d15 test/nvmf: simplify rpc.sh test
Just name the malloc bdev explicitly.  And since there
is only one bdev, remove all of the unnecessary loops.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie3b55161d6ebee64497bc9726edaa3fbf04bb445

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/454690
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-05-17 12:48:19 +00:00