8010 Commits

Author SHA1 Message Date
Konrad Sztyber
6a7c9763e1 bdev/ftl: reduce the number of bdev_ftl_create's params
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id9edced3670f7de1cd08cd8e3481cc5288103458
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449405
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-03-29 04:42:48 +00:00
Ben Walker
c04332d494 rocksdb: Add an SpdkFinalizeThread function
This is just a placeholder. The real implementation comes later.

Change-Id: I657b240977ba09fd6f24e9e36e97e293e73b5801
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449471
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-29 04:42:24 +00:00
paul luse
29b446a1bc bdev/compress: insert vol unload into destruct path
Change-Id: I10c7dff267ce469c0b01db9d6a1dbf89d2fad877
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447368
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-29 04:41:32 +00:00
paul luse
8944bb469f bdev/comrpess: add/fix bdev delete
Was partially present but not complete. Fits here in the series as
it requires read/write operations.

Change-Id: I552896c934bf60256625a83951a95d58d2719646
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447358
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-29 04:41:32 +00:00
paul luse
e2a2b637c9 bdev/compress: incorporate reducelib read and write
Change-Id: I280e9d51bf23ef101b8a3ba8a68a0c1eb10cb65c
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/438274
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-03-29 04:41:32 +00:00
Karol Latecki
02d7812f46 scripts/perf: Update results parsing in nvmf benchmark scripts
Wrong indent level caused results to be duplicated in final
output file if configuration file was used to run more than one
workload combination.

Change-Id: I0fadc10eba38898df648ec2da5fbd0440a3b9693
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449150
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Pawel Kaminski <pawelx.kaminski@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-29 04:41:02 +00:00
Wojciech Malikowski
82645a63d3 test/ftl: Enable extended ftl tests
Change-Id: I073ae6ac435fe7b63ff301a9d49e6b70ee9f148c
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448507
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2019-03-29 04:40:28 +00:00
Ziye Yang
b151999f06 blobfs: Add the return value check for calling cache_append_buffer
Reason: We need to detect this issue. Otherwise, it will
cause null pointer issue later

Change-Id: I4d9aec0570e0b46274ebf3b9642b4727049a4a3b
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448594
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-29 02:38:33 +00:00
Jim Harris
11b38a585a app, log: clarify how to enable log flags
Let's point users to use --enable-debug now instead of
CONFIG_DEBUG=y.  Also to be completely pedantic, use
"configure" instead of "build" to make it more clear
this flag needs to be passed to the configure script,
not make.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4a6f6313a4f8e87cbb1d79cb68645305abb0e106

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449339
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-29 02:35:48 +00:00
Jim Harris
a0070df9f8 nvme/perf: add -G flag to enable debug logging
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I717b738c50ea7b22eb90c6b764a4768a71208990

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449338
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-29 02:35:48 +00:00
Jim Harris
df082c931e reduce: fix chunk_is_compressed calculation
Stupid bug - a chunk is compressed when the number
of io_units for the compressed data does *not* equal
the number of io_units per chunk.

Paul found this during some of his initial DPDK
framework integration testing.  I have some better
unit tests coming up to test this further, but want
to get this obvious fix in for now.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaf9c78a1c1d6045070e7625c6a54ab3d227c8ea7

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449498
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-29 02:34:21 +00:00
Darek Stojaczyk
fb51565a59 env/dpdk: add spdk_pci_fini()
It's mostly needed for the next patch, but even
now it provides some value by printing errors if
there any leaked (still attached) PCI devices
at shutdown.

Change-Id: I8459a6049b3c6612d9f1d99444bf3acfd474a839
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449082
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-28 14:50:46 +00:00
Wojciech Malikowski
04814b72a8 lib/ftl: Internal IO retry mechanism in case ENOMEM from nvme layer
Added internal retry IO queue. In case ENOMEM from
nvme layer request is put to retry queue. When some
inflight operations are completed try to make progress
on IOs in retry queue in first place.

Change-Id: Ie9d3d20bd34431ee57f9454f242b0cdca349c804
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448461
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-28 14:17:11 +00:00
Wojciech Malikowski
f8399d4539 lib/ftl: Update internal IO position in one place
Change-Id: Ie925659e8ffca1fb8aaa1252118bc670b3b274fd
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449015
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-28 14:17:11 +00:00
Wojciech Malikowski
e27f64fb98 lib/ftl: Use ftl_io_done() to determine if IO is completed
Change-Id: I76b58764153793f26b167071041ee17673fec63f
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449160
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-28 14:17:11 +00:00
Darek Stojaczyk
23d7ff31fc vhost: change vsession->lcore only within that lcore
There is currently a small window after we stop
session's pollers and before we mark the session
as stopped (by setting vsession->lcore to -1). If
spdk_vhost_dev_foreach_session() is called within
this window, its callback could assume the session
is still running and for example in vhost scsi
target hotremove case, could destroy an io_channel
for the second time - as it'd first done when the
session was stopped. That's a bug.

A similar case exists for session start.

We fix the above by setting vsession->lcore directly
after starting or stopping the session, hence
eliminating the possible window for data races.

This has a few implications:
 * spdk_vhost_session_send_event() called before
   session start can't operate on vsession->lcore,
   so it needs to be provided with the lcore as
   an additional parameter now.
 * the vsession->lcore can't be accessed until
   spdk_vhost_session_start_done() is called, so
   its existing usages were replaced with
   spdk_env_get_current_core()
 * active_session_num is decremented right after
   spdk_vhost_session_stop_done() is called and
   before spdk_vhost_session_send_event() returns,
   so some active_session_num == 1 checks meaning
   "the last session gets stopped now" needed to be
   changed to check against == 0, as if "the last
   session has been just stopped"

Change-Id: I5781bb0ce247425130c9672e0df27d06b6234317
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448229
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-28 14:16:56 +00:00
Darek Stojaczyk
2cddd571ee vhost: add spdk_vhost_session_start_done/stop_done
Split spdk_vhost_session_event_done() into two separate
functions. This is just a preparation for the next patch.

Change-Id: I05e046e4b963387f058d2b822d7493c761eebbbb
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448228
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-28 14:16:56 +00:00
Darek Stojaczyk
64d76e50cc vhost: call session_event_done() always under the global vhost lock
In the next patch we will put much more responsibility
on spdk_vhost_session_event_done(), so here we make
sure it's always called under the global vhost mutex.

Specifically, spdk_vhost_session_event_done() will set
vsession->lcore, which any other thread might try to
concurrently access via spdk_vhost_dev_foreach_session().

Change-Id: I7a5fde4be4e8bdfdbbb24ac955af964f516bdb68
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448227
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-28 14:16:56 +00:00
Darek Stojaczyk
bbfbadf59a vhost: add spdk_vhost_trylock()
We'll make use of it inside the vhost device backend
code. The function itself is generic enough to be put
in the public vhost.h header rather than vhost_internal.h.

Change-Id: I60602c61d8bba665dcf9c6d27af2e910c208a7be
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448226
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-28 14:16:56 +00:00
yidong0635
0d076e264e bdev/raid: remove unnecessary assert.
Assert(raid_bdev->base_bdev_info) at here is unnecessary.
So remove it.

Change-Id: I7d87fbc95d8c1434ab346fd11179433547b74798
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449379
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-03-28 14:14:38 +00:00
Darek Stojaczyk
6e9eca7874 virtio: switch to spdk_*malloc().
spdk_dma_*malloc() is about to be deprecated.

Change-Id: Iacf9f6536ba5baca7b245e639d0d42a89720ba58
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448173
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-28 07:39:31 +00:00
Jim Harris
2edc652913 reduce: save num_io_units and chunk_is_compressed to req object
When a chunk cannot be compressed enough to save one I/O
unit, we will just write the uncompressed data to disk.  In
this case, we want the helper routine to use the decomp_buf
instead of the comp_buf.

To facilitate that, save some information in the request
object on the number of io_units to be read from or written
to disk, and whether the data is compressed or not.  We
could easily deduce if the data is compressed based on the
number of io_units but saving it in the structure will
save a few instructions and simplify the code a bit.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I1537130577943e19fe4dfcee58eac453eca3c29b

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449259
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-28 03:45:29 +00:00
Jim Harris
309f7791b9 reduce: plumb basic compress/decompress callbacks
The unit tests don't really try to compress anything
yet, but this at least gets the pipeline in place.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ic413850b30e4d9631f3ece2bab40d9026225e5b2

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449097
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-28 03:45:29 +00:00
Jim Harris
72da9b7562 reduce: use consistent callback names
There are a sequence of operations needed for both
read and write I/O.  For example, on a write I/O,
we may need to do a read, modify the chunk, then
write the data back to disk.

For read I/O, the name of the callback is
_read_read_done - the first "read" indicates the
I/O type, the second "read" indicates the disk read
part of the sequence.

The write I/O steps aren't using this same naming
convention, so change that here.  This will be
important in upcoming patches where we'll be adding
compress/decompress steps into the I/O pipeline -
having a consistent naming strategy will make the code
easier to follow.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Icd525874918378118d807acdbe5dd42d5be03126

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449258
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-28 03:45:29 +00:00
Jim Harris
f9ac7dcf11 reduce: move code into _read_complete_req helper function
This preps for some future changes.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I380258938e2412062b43c3d460a7322c5afdf819

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449096
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
2019-03-28 03:45:29 +00:00
Jim Harris
09aff2d0fd reduce: add intermediate buffers for compression operations
We will actually use these in future patches.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I730ef90da6d93922dd17a5ac16594e66132a5f11

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449086
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
2019-03-28 03:45:29 +00:00
Konrad Sztyber
ce95c099a8 ut/ftl: fixed unitialized value warnings
Clear the metadata before using it to avoid valgrind's uninitialized
value warnings.

Change-Id: I1ffd7f50239bace7700d7620b761d1de49942af7
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449158
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-28 02:32:13 +00:00
JinYu
5874e2ac6c nvme:complete I/O and abort rest I/O before destroy io_qpair
Logically we should complete any I/O in the completion queue and
abort the rest of the I/O after we delete the submission queue
and completion queue, so that we would not lost any complete I/O.
We alse should complete I/O and abort I/O before destroy I/O qpair
even though the ctrlr/device has been removed.

Change-Id: Ieb28ad7b4a3a7be553f70178b29ca870b5413191
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449316
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-28 01:06:17 +00:00
Karol Latecki
f8815f02af test/vhost: use spdk/qemu branch v3.0.0 in tests
Change-Id: I7f00bd4c243e07a02ee4a76859e8d890944bab9a
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/437302
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-27 23:56:41 +00:00
Paul Luse
1717b5d580 bdev/compress: add reduce integration
Basic integration, init and load sequences

Change-Id: I12595a251f38168aad15e95275651cb4ce40ea7d
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/435764
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-03-27 19:17:46 +00:00
JinYu
2f3147c0b2 bdev: fix potential segmentation fault bug
As the bdev_io maybe NULL, so put the assignment after assert(bdev_io != NULL).

Change-Id: I58d6fcf97931b2f431ad5680b87b098e0c9a5e9b
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449296
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-03-27 18:38:52 +00:00
Paul Luse
db541f8eb3 bdev/compress: add compression vbdev module
Just the starting point, base functions and structs.

Change-Id: I2ff2b672aa675a19b89e4449f1cc4aa664007f6f
Signed-off-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/435747
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-03-27 14:51:22 +00:00
Jim Harris
20145fd714 reduce: prepend rw to request buf and buf_iov
We will need separate temporary buffers for compress/decompress.
A single temporary buffer won't do because a user's read/write
operations may not cover a full chunk - so we'll need one buffer
to read/write the compressed data, and another buffer for
the uncompressed data.

So for now, just prepend rw to the existing fields - this
will signify these fields are used for the read/write operations
to the backing storage device.  We'll add additional ones in future
patches for the buffers that will hold the uncompressed data.

Note: the vol->buf_mem and ->buf_iov_mem are not prepending "rw"
because they will be used for the compress/decompress temporary
buffers as well.  We'll just double the size of these buffers
when allocating them for the volume.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7df56ab72769f4689f3abac3354446e9cf78d423

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449085
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-27 14:20:26 +00:00
Jim Harris
502ab5b66d reduce: rename vol->reqbufspace to buf_mem
This matches better with the existing buf_iov_mem.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3b85708672553c281666becc8eaf9f9028da8782

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449084
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-27 14:20:26 +00:00
Jim Harris
5a04099b01 reduce: save compressed data size in chunk map
Upcoming patches will try to compress the data before
writing it to backing I/O units, and we'll need to save
the exact number of compressed bytes.

For now, since we're not actually compressing data yet,
just save the uncompressed data size to this field.  There
will be cases when we cannot realize any compression
savings and will just store the uncompressed data, so
handling this now is one less path we'll need to add later.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I499a89590349e4816fe0fef77cc1bfabde6871bf

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449078
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-27 14:20:26 +00:00
Jim Harris
3c070c97ff ut/reduce: remove _vol_get_chunk_map
Just use the new _reduce_vol_get_chunk_map function from
reduce.c instead.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I90e9b05226155d0e8d5e74322b02c669de9204df

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449077
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-27 14:20:26 +00:00
Jim Harris
3bb1f95570 reduce: add struct for chunk maps
Previously we were just passing around uint64_t pointers,
since the chunk map was literally only an array of uint64_t.
But we need to store the compressed data size in the chunk
as well.  Before adding the compressed data size, let's do
the prep work here to just create the chunk map structure
which for now only includes an array of uint64_t.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Icbfa6e509463d4f560444d3db8acd9e87917835f
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449076
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-27 14:20:26 +00:00
Jim Harris
ac3077c3ea test/reduce: add compression algorithm
This is a really dumb algorithm but is simple to understand
for unit testing purposes.  It counts the number of consecutive
bytes with the same value, and then stores the number of bytes
with that value, and the byte value itself.  For example:

0xA1 0xA1 0xA2 0xA3 0xA3 0xA3 0xA3 0xA3 0xA3

would be compressed to:

0x02 0xA1 0x01 0xA2 0x06 0xA3

This will make it easy to generate specific data sequences
that compress to specific sizes to validate the different
compression paths through the reduce library.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0d48f98c080b9cdacb25433f53f44df7ada78c0a

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449072
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-03-27 14:20:26 +00:00
yidong0635
2cc2bbe604 scripts/ceph: reduce ceph_raw.img from 10G to 4G
When testing on VM using a 10G .img file is problematic
due to size - sometimes there is not enough space.
Decreasing the size should improve this issue.

Change-Id: I7be91aaa56a4e7f60cfdffc67f85ae8b3f42b650
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449167
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-03-27 12:03:21 +00:00
Vitaliy Mysak
2c651f101c ocf: batched request processing in queue poller
Process 32 OCF requests during poller invocation.
This will result in more flat events distribution.

This patch uses new API function: ocf_queue_run_single().

Change-Id: I0bc2ef783018fdbcf6f6e938fd7ca136ade61599
Signed-off-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/446635
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2019-03-26 23:30:31 +00:00
Vitaliy Mysak
bef0c6edd2 ocf: rpc: extend get_ocf_bdevs for multicore cases
Extend existing get_ocf_bdevs call to make it easier
  to get information about attached cores.

Implementation is based adding additional optional argument "name"
  to existing call. Based on "name" bdevs are filtered.
  Backward compatability of RPC interface is preserved.

This patch also adds tests for the case when name is given.

Change-Id: I4300ebe37e936bc5cca8e066b5f09db462a87cf7
Signed-off-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/444841
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2019-03-26 23:30:31 +00:00
Darek Stojaczyk
530f481259 blobstore: switch to spdk_*malloc().
spdk_dma_*malloc() is about to be deprecated.

Change-Id: I0da485ec68329ee9f819cf6dfdb8744c5993e6f2
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448171
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-03-26 23:27:29 +00:00
Chunyang Hui
38902a5a27 rpc: add get_spdk_version rpc method
Change-Id: Iacc1766a60fb1e15a1013475f83f7f13cd45fe12
Signed-off-by: Chunyang Hui <chunyang.hui@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448613
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-26 22:54:28 +00:00
Michal Mielewczyk
b949f1318e ocf: Added zeroing memory returned from mempool
OCF requires memory retrived from mempool to be filled with zeros.

When mempool is created, information about element size is stored to be used
when new memory is retrived.

This is fix for issue #671

Signed-off-by: Michal Mielewczyk <michal.mielewczyk@intel.com>
Change-Id: Ieb533e3bdae0665dae18e7b3a379da0ed843c35a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449155
Reviewed-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-26 22:53:09 +00:00
Karol Latecki
b6abc16b05 test: remove duplicate fio.py script file
scripts/fio.py and test/nvmf/fio/nvmf_fio.py are almost exact
same script. This commit unifies two files into one.

Change-Id: If753baaeb9f92dad2cda27bb4bed78ade4827d0b
Signed-off-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448656
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Vitaliy Mysak <vitaliy.mysak@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2019-03-26 22:47:45 +00:00
Konrad Sztyber
02b0230296 bdev/ftl: treat null UUID as no UUID
Check if the UUID in the configuration is null (all zeroes) and treat it
as if no UUID has been supplied.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ic91c77591528a8aaa4cf5c0241e6bde51b3757f1
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448628
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-26 12:01:48 +00:00
Konrad Sztyber
9674a7b6cb bdev/ftl: better config errors logging
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ie3814a905efa6122d56a5f2dfc9710d016e8449e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448713
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-26 12:01:48 +00:00
Shuhei Matsumoto
3c4199d6c6 env_dpdk: Run DPDK in legacy memory mode through spdk_env_opts
Some customized SPDK application needs DPDK to run legacy memory mode,
but it is  unusual. So this patch uses env_context, the opaque option
variable in the struct spdk_env_opts, and if the application sets
"--legacy-mem" to it, spdk_env_opts_init() skips adding
"--match-allocations" for DPDK 19.02 or later and skips adding
"--legacy-mem" for DPDK 18.05 or before.

Change-Id: I6f40c726c66c29f0aabfeeaecd00611954dc774f
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448263
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-03-26 12:00:36 +00:00
Pawel Kaminski
8228a208a6 test/vhost: Reduce time needed to wait for vm boot in vhost tests.
Help to fix #695

Change-Id: I102ab6f7a19c50a128899178517d323f62a39945
Signed-off-by: Pawel Kaminski <pawelx.kaminski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449008
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-03-26 11:26:31 +00:00
Pawel Kaminski
25dbb5f837 test/vhost: Print more logs from VM
Help to fix #695.

Change-Id: I124ec865711b45df031c92fc6d52037047bd1006
Signed-off-by: Pawel Kaminski <pawelx.kaminski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448575
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-03-26 11:26:31 +00:00