Commit Graph

9001 Commits

Author SHA1 Message Date
Wojciech Malikowski
110906e8a8 test/blobfs/fuse: Optional VMD enumeration
VMD section with Enable flag set to true
need to be defined in config file to
enumerate devices behind VMD.

Change-Id: Ib4d6fad99979d2e8ab61e18feb04661f388eb4c1
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459762
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:53:06 +00:00
Wojciech Malikowski
1ee1acfea1 app/iscsi_tgt: Optional VMD enumeration
VMD section with Enable flag set to true
need to be defined in config file to
enumerate devices behind VMD.

Change-Id: Ie1cb67f7e58d393783eeac6b0e437ee36bdeb0d9
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459643
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
2019-07-01 03:53:06 +00:00
Wojciech Malikowski
fd508c14ca app/vhost: Optional VMD enumeration
VMD section with Enable flag set to true
need to be defined in config file to
enumerate devices behind VMD.

Change-Id: Id22e0b940e789a32507e582efda564adae5beb28
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459642
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:53:06 +00:00
Wojciech Malikowski
54020194c6 app/nvmf_tgt: Optional VMD enumeration
VMD section with Enable flag set to true
need to be defined in config file to
enumerate devices behind VMD.

Change-Id: Idbed36e57e3987486d8f0fb5a6c6336b4679344a
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459641
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:53:06 +00:00
Wojciech Malikowski
f5e121fee0 app/bdev_svc: Optional VMD enumeration
VMD section with Enable flag set to true
need to be defined in config file to
enumerate devices behind VMD.

Change-Id: If0bf702dcdc19bfaec44fb8867205f29f562097e
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459640
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:53:06 +00:00
Wojciech Malikowski
e2fb1b80e1 lib/ftl: Check if any additional relocation was added
Return immediatly from ftl_reloc_add() if no
new blocks was added to relocation.

Change-Id: If80dfa725e0bb9f3b8987740012858a671c5ad90
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457626
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
a74b79dc74 lib/ftl: Drop relocation for empty bands immediately
Added check if band that is added to reloc
have any valid blocks. Return immediately
if there is no valid blocks.

Change-Id: I2bce088e0ad71479c6899fff96845397d12e2e92
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457625
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
c149f9597f lib/ftl: Prevent from adding active reloc to pending queue
In case ANM event occurs on band being
relocating (band is on active reloc queue)
we shoudn't add such band to pending queue.

Change-Id: I92a8bee11309097e19afaea549460f1d4387e3e5
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458617
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
76cff6da81 lib/ftl: Remove band from active/pending queue
In case high priority band was added
for relocation it should be removed
from active/pending queue if it was
already on one of them.

Change-Id: Id0591b1d3a4174dd05eb1c32227e4d3b3a9cbcd0
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458057
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
2cc6bd2a26 lib/ftl: Skip block with ongoing write during relocation
In case ANM event occurs on open band there
can be situation that reloc will try to read
block on which there is ongoing write.
This is happening because lba valid map is
updated before write  submission to allow sent
consistent metadata to disk before all user
writes are completed.

Added write offset to the each chunk and
add check to reloc if particular ppa is written
on that chunk.

Change-Id: Ic95a06e69381d2152a86984b65a0975afaff955d
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458056
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
bf4973087f lib/ftl: Allow for relocating open band
In case ANM event occurs on open band
reloc need to be able to process such
event.

If band is not in closed sate do not
alloc lba map for it and do not set
it to free state after relocation.

Change-Id: I2f4a5770fef08271d222936ca19f3cc98e5e5be1
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457612
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
f8a9112292 lib/ftl: Mark all lba map segments as cached for open bands
Open bands need to have lba map segments
set to cached state to prevent read lba
map from disk during relocation events.

Change-Id: Ib4f1ed19131fad174c1d2f70e4c02e83701e2a0a
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457853
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
bfd67f9405 lib/ftl: Initialize band tail metadata physical address
Band tail PPA should be initialized when
new FTL instance is created.

Change-Id: Ie2fb72aa3f29eece0b6f8912998b33af3ba6b355
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457777
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-07-01 03:29:00 +00:00
Wojciech Malikowski
4be37a57f4 lib/ftl: Consume ANM event on core thread
Send ANM event to core thread for further
processing. This will remove a need of locking
in relocate module when ANM event occur.

Change-Id: I0efb1f1b8c96c107cda5fe78e8ee5672cde39f11
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457611
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2019-07-01 03:29:00 +00:00
Ziye Yang
cdc0170c1b nvmf/tcp: Add a maximal PDU loop number
In our previous code, we will handle all the PDU until there is
no incoming data from the network if we can continue the loop.
However this is not quite fair when we handling multiple connections
in a polling group.

And this change is setting a maximal NVME/TCP PDU we can handle
for each conneciton, it can improve the performance. After some
tuing, 32 should be a good loop number. Our iSCSI target uses
16.

The following shows some performance data:

Configuration:
1 Command used in the initiator side:
./examples/nvme/perf/perf -r 'trtype:TCP adrfam:IPv4 traddr:192.168.4.11 trsvcid:4420'
-q 128 -o 4096 -w randrw -M 50 -t 10

2 target side, export 4 malloc bdev in a same subsystem

Result:

Before patch:

Starting thread on core 0
========================================================
                                                                                                           Latency(us)
Device Information                                                    :       IOPS      MiB/s    Average        min        max
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   51554.20     201.38    2483.07     462.31    4158.45
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   51533.00     201.30    2484.12     508.06    4464.07
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   51630.20     201.68    2479.30     481.19    4120.83
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   51700.70     201.96    2475.85     442.61    4018.67
========================================================
Total                                                                 :  206418.10     806.32    2480.58     442.61    4464.07

After patch:
Starting thread on core 0
========================================================
                                                                                                           Latency(us)
Device Information                                                    :       IOPS      MiB/s    Average        min        max
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   57445.30     224.40    2228.46     450.03    4231.23
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   57529.50     224.72    2225.17     676.07    4251.76
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   57524.80     224.71    2225.29     627.08    4193.28
TCP  (addr:192.168.4.11 subnqn:nqn.2016-06.io.spdk:cnode1) from core 0:   57476.50     224.52    2227.17     663.14    4205.12
========================================================
Total                                                                 :  229976.10     898.34    2226.52     450.03    4251.76

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I86b7af1b669169eee2225de2d28c2cc313e7d905
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459572
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-28 12:28:54 +00:00
Hailiang Wang
5b80a5505c test/iscsi_ut: fix a warning of rsp_pdu garbage value
Compilation Warning on fedora30.
iscsi_ut.c:184:31: warning: The left operand of '==' is a garbage value
        CU_ASSERT(rsph->status_class == ISCSI_CLASS_INITIATOR_ERROR);
                  ~~~~~~~~~~~~~~~~~~ ^
This is related to issue #822.

Change-Id: I221aa6601bae4a2d55bd557fcd1d001fbc616149
Signed-off-by: Hailiang Wang <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459723
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-28 12:26:51 +00:00
Hailiang Wang
94c06a68de test/dif_ut: fix a warning of iov_base garbage value
Compilation Warning on fedora30.
dif_ut.c:151:24: warning: The left operand of '==' is a garbage value
        return (iov->iov_base == iov_base && iov->iov_len == iov_len);
                ~~~~~~~~~~~~~ ^
This is related to issue #822.

Change-Id: Ie050c65d3747c551e2c26e84d3879742eaf86fd8
Signed-off-by: Hailiang Wang <hailiangx.e.wang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459712
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-28 12:26:51 +00:00
Tomasz Zawadzki
bef870c6d0 test/fio: fail the tests when fio is not available
When possible we should make sure that no silent failures or
skipped tests.
Since some of the tests use fio, autotest should fail
in those places when it is missing.

Change-Id: I97581e2df30a02445b560e3b88d03ce92986d0fd
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456316
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-28 12:26:12 +00:00
Changpeng Liu
6c9b6abf5e blobfs: make internal asynchronous APIs as public APIs
SPDK blobfs has asynchronous APIs defined in blobfs_internal.h file,
as users may want to use them, so we remove them to the public .h file.

Change-Id: I1835d97060101f6315a73cb8638b15ff7e13ba54
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457547
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-28 09:50:50 +00:00
Changpeng Liu
485a2dcfa9 UT/blobfs: add asynchronous Writev/Readv unit tests
Change-Id: I8c0961e78afff367f6fc8b1a97d5ec5c635e3224
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457455
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-28 09:50:50 +00:00
Changpeng Liu
1966f1eef3 blobfs: add writev/readv asynchronous APIs support
Change-Id: Id1172f546852fcf25c6d13cb63f9d875b02e768c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/453493
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-28 09:50:50 +00:00
Takeshi Yoshimura
c74ea9fa8e rocksdb: Fix null deref at blobfs calls
I tried experimental binding of SPDK with Mongo-rocks. However,
the binding sometimes invoke blobfs APIs without thread
initializations. In that case, null dereferences occur. In other
words, we need to carefully use blobfs not to invoke any threads
that are not registered to blobfs.

This patch simply adds a sanity check at every use of blobfs APIs.
By doing this, we do not need to care about which threads can use
blobfs APIs.

Change-Id: I5b37b0267306a7c76d20e81c1773a6a33be7828c
Signed-off-by: Takeshi Yoshimura <t.yoshimura8869@gmail.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/418966
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-28 08:37:54 +00:00
Anil Veerabhadrappa
9d1e666798 conf: parse "C2HSuccess" parameter for TCP transport only
"C2HSuccess" is only valid for TCP transport. So this parameter
should be looked up only for TCP transport. Without the change,
spdk_nvmf_parse_transport() would bailout early for RDMA and
other transports without every creating them.

Signed-off-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Change-Id: I34bdff2f4ab930516743cd5dbf022d75e60fd85c
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459571
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-28 07:03:45 +00:00
Liang Yan
4ebf5cef63 perf/nvmf: add link files into zipped SPDK source codes
Change-Id: I9581c936a40be8ccf352778c62bba09ab5a9428e
Signed-off-by: Liang Yan <liang.z.yan@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459091
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-28 05:36:37 +00:00
Ziye Yang
fd22bfcc2b iscsi/test: destruct all malloc devices by one time
Just use a for loop, then we can delete all malloc
devices.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Change-Id: I34fb06c0d49333ab1b5bce0ea2e2b7ced7c50eb2
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458045
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-28 05:28:09 +00:00
Shuhei Matsumoto
0c26ea5a2b dif: Factor out converting size from LBA based to extended LBA based
In DIF library there are many functions that converts offset or
length from LBA based to extended LBA based.

Factor out them by adding a helper function _to_size_with_md().

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Id5576edacc8a07095726f659c4b53ac3aa83727d
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459530
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-06-28 04:13:02 +00:00
Shuhei Matsumoto
1d1c60e53d dif: Add helper function to convert buffer range to extended LBA based
This will be used to get extended LBA based range or length in NVMe/TCP
target later.

Change-Id: Id0f08bdaeea634dbc05b34a0f7914be21aef9aae
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458706
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-28 04:13:02 +00:00
Shuhei Matsumoto
767c046e77 dif: Add spdk_dif_update_crc32c_stream to update CRC32C by stream fashion
Add spdk_dif_update_crc32c_stream to update CRC32C by stream fashion.
spdk_dif_update_crc32c_stream utilizes the updated _dif_update_crc32c_split.

A minor bug was found in UT for spdk_dif_update_crc32c and is fixed
together in this patch.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I92358e845e8e2e17c6f288aa718b947e71e6e1fb
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458919
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-06-28 04:13:02 +00:00
Shuhei Matsumoto
4c2e52b935 dif: Process partial data block properly in _dif_update_crc32c_split
For NVMe/TCP target, data segments which correspond to H2C or C2H PDU
will have any alignment, and _dif_update_crc32c_split will have to
process partial data block, particularly the following types:
- start and end are both within a data block.
- start is within a data block, and end is at the end of a block

On the other hand, _dif_update_crc32c_split had assumed that passed
block is always a complete block.

This patch exposes offset_in_block, data_len, and guard as parameters
of _dif_update_crc32c_split() and make _dif_verify_split() process
the above two types of data block properly.

The next patch will utilize the updated _dif_update_crc32c_split to
add spdk_dif_update_crc32c_stream().

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Iee29377ad49d4f209673fffb4de4a23a54f31766
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458918
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2019-06-28 04:13:02 +00:00
Wojciech Malikowski
c3abcfefda app/spdk_lspci: App for printing available SPDK PCI devices
This application enumerates all PCI devices
in system (including VMD) and prints their
address, vendor ID and device ID.

Change-Id: I2e73bc62c0a282b673486fa4527ff4b8319d47a2
Signed-off-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/450925
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-27 12:20:35 +00:00
yidong0635
310fc0b5d5 scsi_bdev_ut: add SPDK_CU_ASSERT_FATAL before put task.
Like other test, need to add SPDK_CU_ASSERT_FATAL(TAILQ_EMPTY(&g_bdev_io_queue)).
To avoid issue:
scsi_bdev_ut.c:956:2: warning: Address of stack memory associated with local
variable 'task' is still referred to by the global variable 'g_io_wait_queue'
 upon returning to the caller.  This will be a dangling reference
        ut_put_task(&task);
        ^~~~~~~~~~~~~~~~~~
waiting exit from while loop in ut_bdev_io_flush.

This is related to issue #822.

Change-Id: If390f5076582a0149646f0c2a408c3bcd3b4fab6
Signed-off-by: yidong0635 <dongx.yi@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459534
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 12:09:56 +00:00
Tomasz Kulasek
9c9709b9b1 test/iscsi_tgt: run vpp with gdb
Change-Id: Ibffdc0d198451d7ef891b4062205c5e0eaf79bee
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457471
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 08:23:08 +00:00
Tomasz Zawadzki
ef489303b4 net/vpp: include VPP into iSCSI test scripts
Tests that were thus far performed using posix net framework
can now be run with VPP. This patch adds network interface
configuration for VPP to work in iSCSI tests.

Some tests are disabled on purpose:
- IP Migration, RBD and NVMe-oF due to their tests lacking network
  namespace support
- rpc_config adding/deleting IP, as VPP has separate utility for that

calsoft.sh doesn't handle TCP stream properly and fails decoding iSCSI
requests when are divided by TCP segmentation. This is very common
situation for VPP and causes that calsoft.sh never PASS.

Change-Id: I7c80427ca1675a1789ce7440796cc8d9956f1c9e
Signed-off-by: Slawomir Mrozowicz <slawomirx.mrozowicz@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/394174
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 08:23:08 +00:00
Tomasz Kulasek
29408ebaa9 test/vm_setup: install VPP 19.01.1
NOTE:

Due to some changes in VPP two cherry-picks from newer version of VPP
(at least 19.04):

	"sock api: allow to start client with no rx_thread"
		commit: 97dcf5bd26ca6de580943f5d39681f0144782c3d

	"dlmalloc: honor 8 byte alignment requests"
		commit: f5dc9fbf814865b31b52b20f5bf959e9ff818b25

These commits are already merged in VPP 19.04.

This patch includes also workaround for VPP 19.01.1 to prevent closing
sessions to already closed applications after timeout. It causes
intermittent segfaults.
This is temporal solution and should be solved with next releases of
VPP, but is required now to create more stable VPP environment.

Change-Id: If4b45b7159819cfd836dd7d50f333dbab2b38eab
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456462
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2019-06-27 08:23:08 +00:00
Tomasz Kulasek
338d4a2611 doc/vpp: update VPP integration docs
Change-Id: Ie3ed7362bcc5effd82403565f110decb9ff6fb3a
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456503
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2019-06-27 08:23:08 +00:00
Tomasz Zawadzki
699a5f35e5 net/vpp: switch to session.api
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I94ce2735c3d15dd7ee5e4ad33280e9996740e244
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/417056
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-27 08:23:08 +00:00
Tomasz Kulasek
14a62d67ac net/vpp: allow to compile with vpp
This patch prepares environment to allow compile applications and
libraries with VPP.

Change-Id: Icc067c17b57932062afa5e6a67791bd58de3a2cc
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456461
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
2019-06-27 08:23:08 +00:00
Tomasz Kulasek
a44c743417 test/iscsi_tgt: make sure iscsi devices are ready
We have no guaranty that iscsiadm -m session shows all required iscsi
devices at some point of time after login. It may cause that not all
devices are used in further tests (e.g. in fio).

This patch allows to wait for a certain number of sessions after login.

Change-Id: I868cb8f1235dd04d88572bc8518e97de99a2c5e6
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458236
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 08:23:08 +00:00
Andrey Kuzmin
0972838f83 bdevperf: provide unregister callback when opening target bdev.
Bdevperf up to now didn't use unregister callback when opening target
block deice. This is unsafe if test scenario involves target bdev
unregistering voluntarily, although with recent bdev layer update
not registering hotremove callback has become safer.

To be on the safe side, this patch registers a callback to transit
gone target into the I/O drain mode in case the target unregisters
during runtime.

Changes wrt previous versions of the patch:

	- incorporated review comments
	- updated callback functionality
	- updated patch description.

Change-Id: I0dad2f4a7cf4acb22b363e23e4b670ca73d8c6f2
Signed-off-by: Andrey Kuzmin <akuzmin@jetstreamsoft.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/447956
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 06:16:11 +00:00
Darek Stojaczyk
f9a6588f57 nvme: switch to spdk_*malloc().
spdk_dma_*malloc() is about to be deprecated.

Change-Id: I6c308ee546c28c479ceb903bc1749bf5209dc6fe
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/448172
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: <uma.willpower@gmail.com>
2019-06-27 04:34:50 +00:00
Shuhei Matsumoto
a019feec62 dif: Factor out setup operation of spdk_dif_generate/verify_stream
spdk_dif_generate_stream() and spdk_dif_verify_stream() are very
similar. Factoring out the common part into a function will improve
the maintainability and do in this patch.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I16ecd0860c75037d9182298d7513749dfe8e9b56
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458376
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 00:36:25 +00:00
Shuhei Matsumoto
6db126c24c dif: Add spdk_dif_verify_stream to verify DIF by stream fashion
Add spdk_dif_verify_stream to verify DIF by stream fashion.
spdk_dif_verify_stream utilizes the updated _dif_verify_split.
spdk_dif_verify_stream is very similar with spdk_dif_generate_stream().

UT code demonstrates how it is realized.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I1c5d197cf4c0bbc82c8e7f4fa45ddc0b94051058
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458330
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 00:36:25 +00:00
Shuhei Matsumoto
80f2ca0d90 dif: Process partial data block properly in _dif_verify_split
For NVMe/TCP target, data segments which correspond to H2C or C2H PDU
will have any alignment, and _dif_verify_split will have to process
partial data block, particularly the following types:
- start and end are both within a data block.
- start is within a data block, and end is at the end of a block

On the other hand, _dif_verify_split had assumed that passed block
is always a complete block.

According to the refactoring done in the last patch, this patch
exposes offset_in_block, data_len, and guard as parameters of
_dif_verify_split() and make _dif_verify_split() process the
above two types of data block properly.

The next patch will utilize the updated _dif_verify_split to
add spdk_dif_verify_stream().

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Ic371d3ccefbd5fe8147a948a624013be2702128e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458329
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 00:36:25 +00:00
Shuhei Matsumoto
d69dc28b00 dif: Separate _dif_verify_split into three parts
For NVMe/TCP target, data segments which correspond to H2C or C2H PDU
will have any alignment, and _dif_verify_split will have to process
partial data block, particularly the following types:
- start and end are both within a data block.
- start is within a data block, and end is at the end of a block

On the other hand, _dif_verify_split had assumed that passed block
is always a complete block.

To process the above types, separating guard computation, DIF copy
and skipping metadata field, and  DIF verification into three parts
will be helpful and is done in this patch.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Ic4f1765e01507efa812dfaf7a8018666c6346f8e
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458328
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-27 00:36:25 +00:00
Konrad Sztyber
42287a6954 lib/ftl: error logging around getting chunk info
Some of the errors were silent, making it hard to pinpoint the exact
failing call. This patch adds SPDK_ERRLOGs for each error path.

Change-Id: I71be6c97cab916ac52314e5f4e4d63358877bd96
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458426
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
2019-06-26 14:21:52 +00:00
Konrad Sztyber
18b1de97d8 lib/ftl: store metadata on non-volatile cache
Send LBA along with the data block when mirroring writes to the
non-volatile cache. The metadata buffer is retrieved from the metadata
pool, so the maximum number of concurrent requests is limited to
nv_cache.max_request_cnt, while the number of blocks in a single request
is limited by nv_cache.max_requets_size.

Change-Id: If260302d16039183fb0fe073ef7419947532cfab
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458093
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@intel.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-26 09:36:57 +00:00
Konrad Sztyber
11ff1f4a2b lib/ftl: non-volatile cache metadata pool
Initialize the memory pool for storing metadata (LBAs) when writing data
to the non-volatile cache. The mempool's object count and size can be
configured via nv_cache.max_request_cnt / nv_cache.max_request_size
respectively.

Change-Id: I376df9a75be13d4b29ba475f350edf402c868d48
Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/458092
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
2019-06-26 09:36:57 +00:00
JinYu
83f3785bc4 test: fix the nvme reset test time
As have updated the NVMe reset case, the check_io() must be executed
after ctrlr reset. It's easier to reproduce #819, so we can set a
short time.

Change-Id: I9ba867a6c2aa1a3921bace2e77acd28a069d9016
Signed-off-by: JinYu <jin.yu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457761
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-26 08:39:14 +00:00
Mateusz Kozlowski
8c2cff02b6 lib/ftl: Fix ppa pack function
Address translation wasn't correct for >32 bit length packed address.
This commit fixes the issue and adds a corresponding unit test.

This patch fixes issue #774:
https://github.com/spdk/spdk/issues/774

Signed-off-by: Mateusz Kozlowski <mateusz.kozlowski@intel.com>
Change-Id: Idce67c47f2a9888f9e2ae2eadaf71ccc34e5c260
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457114
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Wojciech Malikowski <wojciech.malikowski@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2019-06-26 08:38:32 +00:00
John Kariuki
851830c366 nvme/perf: add lto
Updated doc to include section on how to compile SPDK with link
time optimization (lto) gcc flag prior to running the benchmark.

Signed-off-by: John Kariuki <John.K.Kariuki@intel.com>
Change-Id: I1af4033144dd9f7594bcf40621acb7fb52018e7a
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/459330
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2019-06-26 08:25:02 +00:00