SPDK 19.10.1

Change-Id: I3aa4fadc32a381cfe0686650db8bf409f0834ae0
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/478449
Community-CI: SPDK CI Jenkins <sys_sgci@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Alexey Marchuk <alexeymar@mellanox.com>

CHANGELOG.md

@@ -1,6 +1,38 @@
 # Changelog
 
-## v19.10: (Upcoming Release)
+## v19.10.1:
+
+### thread
+
+`spdk_thread_send_msg` now returns int indicating if the message was successfully
+sent.
+
+### dpdk
+
+DPDK submodule updated to include fix for vhost CVE-2019-14818.
+
+## v19.10:
+
+### rpc
+
+Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release.
+The old names will continue to function, but will display a deprecation warning.
+
+Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to
+'bdev_nvme_set_options' RPC method.
+
+Added optional parameter '--md-size' to 'construct_null_bdev' RPC method.
+
+Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev'
+RPC method.
+
+Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev.
+
+Added `blobfs_create` RPC method to build blobfs on given bdev.
+
+Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE.
+Then on the host path, user can directly do some file operations which will be mapped
+to blobfs.
 
 ### bdev
 
@@ -9,6 +41,42 @@ Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and
 the NVMe completion queue DW0 entry. This allows vendor specific IO commands
 to return commmand specific completion info back to the initiator.
 
+Added `spdk_bdev_get_write_unit_size()` function for retrieving required number
+of logical blocks for write operation.
+
+New zone-related fields were added to the result of the `get_bdevs` RPC call:
+ - `zoned`: indicates whether the device is zoned or a regular
+   block device
+ - `zone_size`: number of blocks in a single zone
+ - `max_open_zones`: maximum number of open zones
+ - `optimal_open_zones`: optimal number of open zones
+The `zoned` field is a boolean and is always present, while the rest is only available for zoned
+bdevs.
+
+A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated.
+The new open function introduces requirement to provide callback function that will be called by
+asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as
+an argument instead of bdev structure to avoid a race condition that can happen when the bdev
+is being removed between a call to get its structure based on a name and actually openning it.
+
+New 'resize' event has been added to notify about change of block count property of block device.
+Event is delivered only if block device was opened with `spdk_bdev_open_ext` function.
+
+### bdev zone
+
+Added new public header for zoned bdev. Zoned bdev is an extension
+of the bdev interface.
+
+`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()`
+APIs were added for retrieving zoned device information.
+`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned
+device.
+Added `spdk_bdev_zone_management()` API for changing zone state.
+`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for
+appending data to a zone.
+Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O.
+Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics.
+
 ### bdev opal
 
 EXPERIMENTAL: A new opal bdev has been added to support management of
@@ -23,102 +91,22 @@ It does not yet support recreating the opal bdevs after application restart.
 This bdev module should be considered very experimental, and the RPCs may
 change significantly in future releases.
 
-### bdev zone
+### delay bdev
 
-Added new public header for zoned bdev. Zoned bdev is an extension
-of the bdev interface.
+The `bdev_delay_update_latency` has been added to allow users to update
+a latency value for a given delay bdev.
 
-`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()`
-APIs were added for retrieving zoned device information.
-`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned
-device.
-Added `spdk_bdev_zone_management()` API for changing zone state.
-`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for
-appending data to a zone.
-Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O.
+### compress bdev
 
-### bdev
+A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs
+that were not loaded due to a missing pm metadata file. In this state they
+can only be deleted.
 
-Added `spdk_bdev_get_write_unit_size()` function for retrieving required number
-of logical blocks for write operation.
+### null bdev
 
-New zone-related fields were added to the result of the `get_bdevs` RPC call:
- - `zoned`: indicates whether the device is zoned or a regular
-   block device
- - `zone_size`: number of blocks in a single zone
- - `max_open_zones`: maximum number of open zones
- - `optimal_open_zones`: optimal number of open zones
-The `zoned` field is a boolean and is always present, while the rest is only available for zoned
-bdevs.
+Metadata support has been added to Null bdev module.
 
-### nvmf
-
-The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts`
-as its only parameter. This new structure contains the max_subsystems parameter previously
-passed into that function.
-
-A new public API function `spdk_nvmf_get_tgt` has been added which allows users to
-retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special
-case where an RPC or application only creates a single target, this function can accept
-a null name parameter and will return the only available target.
-
-The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will
-allow those RPCs to work with applications that create more than one target.
-
-Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and
-`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target
-objects. In SPDK the target object defines a unique discovery service. As of this release,
-these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and
-nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py
-
-Three new header functions have also been added to help deal with multiple targets.
-`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name.
-`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list.
-`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list.
-
-The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context
-information to the `new_qpair` callback. This will simplify the code when having multiple
-nvmf targets or when retrieving the context information from globals is not suitable.
-
-### bdev
-
-A new spdk_bdev_open_ext function has been added and spdk_bdev_open function has been deprecated.
-The new open function introduces requirement to provide callback function that will be called by
-asynchronous event such as bdev removal. spdk_bdev_open_ext function takes bdev name as
-an argument instead of bdev structure to avoid a race condition that can happen when the bdev
-is being removed between a call to get its structure based on a name and actually openning it.
-
-New 'resize' event has been added to notify about change of block count property of block device.
-Event is delivered only if block device was opened with spdk_bdev_open_ext function.
-
-### blobstore
-
-A new spdk_bdev_create_bs_dev_from_desc function has been added and spdk_bdev_create_bs_dev
-function has been deprecated.
-The new create function can cowork with spdk_bdev_open_ext function, which provides callback
-function that will be called by asynchronous event such as bdev removal.
-
-### DPDK
-
-Updated DPDK submodule to DPDK 19.08.
-
-### blobfs_bdev
-
-A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev.
-
-Function spdk_blobfs_bdev_detect is added to detect whether blobfs exists on the given block device.
-
-Function spdk_blobfs_bdev_create is added to create a blobfs on the given block device.
-
-Function spdk_blobfs_bdev_mount is added to mount a blobfs on the given block device to
-a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle
-FUSE request by blobfs API.
-
-### build
-
-Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem.
-It requires the installation of libfuse3. By default, it is disabled. And it will be
-enabled if run `./configure` with `--with-fuse` option.
+Protection information support has been added to Null bdev module.
 
 ### nvme
 
@@ -156,6 +144,64 @@ applications when a qpair is failed. This list of functions includes:
 These functions now return -ENXIO when the qpair or controller on which they
 operate is failed.
 
+EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux
+kernel for controller as well as for namespace and process ioctl requests as usual
+from linux environment.
+
+### nvmf
+
+The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts`
+as its only parameter. This new structure contains the max_subsystems parameter previously
+passed into that function.
+
+A new public API function `spdk_nvmf_get_tgt` has been added which allows users to
+retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special
+case where an RPC or application only creates a single target, this function can accept
+a null name parameter and will return the only available target.
+
+The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will
+allow those RPCs to work with applications that create more than one target.
+
+Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and
+`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target
+objects. In SPDK the target object defines a unique discovery service. As of this release,
+these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and
+nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py
+
+Three new header functions have also been added to help deal with multiple targets.
+`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name.
+`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list.
+`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list.
+
+The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context
+information to the `new_qpair` callback. This will simplify the code when having multiple
+nvmf targets or when retrieving the context information from globals is not suitable.
+
+### blobstore
+
+A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev`
+function has been deprecated.
+The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback
+function that will be called by asynchronous event such as bdev removal.
+
+### blobfs_bdev
+
+A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev.
+
+Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device.
+
+Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device.
+
+Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to
+a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle
+FUSE request by blobfs API.
+
+### build
+
+Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem.
+It requires the installation of libfuse3. By default, it is disabled. And it will be
+enabled if run `./configure` with `--with-fuse` option.
+
 ### iSCSI
 
 Portals may no longer be associated with a cpumask. The scheduling of
@@ -165,45 +211,23 @@ An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authenticati
 for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites
 the setting by the global parameters for the iSCSI portal group.
 
-### delay bdev
+### socket
 
-The `bdev_delay_update_latency` has been added to allow users to update
-a latency value for a given delay bdev.
+Added `spdk_sock_is_connected` to check whether the socket is currently connected.
+`spdk_sock_group_poll` now returns number of events on success.
 
-### compress bdev
+### env
 
-A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs
-that were not loaded due to a missing pm metadata file. In this state they
-can only be deleted.
-
-### null bdev
-
-Metadata support has been added to Null bdev module.
-
-Protection information support has been added to Null bdev module.
+Added `spdk_pci_device_unclaim()` function to cleanup pci claim file.
 
 ### event
 
-start_subsystem_init RPC no longer stops the application on error during
+`framework_start_init` RPC no longer stops the application on error during
 initialization.
 
-### rpc
+### DPDK
 
-Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to
-'bdev_nvme_set_options' RPC method.
-
-Added optional parameter '--md-size' to 'construct_null_bdev' RPC method.
-
-Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev'
-RPC method.
-
-Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev.
-
-Added `blobfs_create` RPC method to build blobfs on given bdev.
-
-Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE.
-Then on the host path, user can directly do some file operations which will be mapped
-to blobfs.
+Updated DPDK submodule to DPDK 19.08.
 
 ### ocf
 

doc/bdev.md

@@ -411,6 +411,23 @@ To remove an NVMe controller use the bdev_nvme_detach_controller command.
 
 This command will remove NVMe bdev named Nvme0.
 
+## NVMe bdev character device {#bdev_config_nvme_cuse}
+
+This feature is considered as experimental.
+
+Example commands
+
+`rpc.py bdev_nvme_cuse_register -n Nvme0 -p spdk/nvme0`
+
+This command will register /dev/spdk/nvme0 character device associated with Nvme0
+controller. If there are namespaces created on Nvme0 controller, for each namespace
+device /dev/spdk/nvme0nX is created.
+
+Cuse devices are removed from system, when NVMe controller is detached or unregistered
+with command:
+
+`rpc.py bdev_nvme_cuse_unregister -n Nvme0`
+
 # Logical volumes {#bdev_ug_logical_volumes}
 
 The Logical Volumes library is a flexible storage space management system. It allows

doc/img/nvme_cuse.svg

@@ -0,0 +1,124 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<svg width="193.94mm" height="139.71mm" version="1.1" viewBox="0 0 193.94 139.71" xmlns="http://www.w3.org/2000/svg" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
+ <title>NVMe CUSE</title>
+ <defs>
+  <marker id="marker9353" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker7156" overflow="visible" orient="auto">
+   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker4572" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker4436" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker4324" overflow="visible" orient="auto">
+   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker2300" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker2110" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker2028" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker1219" overflow="visible" orient="auto">
+   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="Arrow1Lstart" overflow="visible" orient="auto">
+   <path transform="matrix(.8 0 0 .8 10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="marker1127" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+  <marker id="Arrow1Lend" overflow="visible" orient="auto">
+   <path transform="matrix(-.8 0 0 -.8 -10 0)" d="m0 0 5-5-17.5 5 17.5 5z" fill-rule="evenodd" stroke="#000" stroke-width="1pt"/>
+  </marker>
+ </defs>
+ <metadata>
+  <rdf:RDF>
+   <cc:Work rdf:about="">
+    <dc:format>image/svg+xml</dc:format>
+    <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
+    <dc:title>NVMe CUSE</dc:title>
+   </cc:Work>
+  </rdf:RDF>
+ </metadata>
+ <g transform="translate(-2.1066 -22.189)">
+  <rect x="11.906" y="134.85" width="72.004" height="20.6" ry="3.7798" fill="none" stroke="#000" stroke-width=".5"/>
+  <text x="14.363094" y="149.02231" fill="#000000" font-family="sans-serif" font-size="10.583px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px" style="line-height:1.25" xml:space="preserve"><tspan x="14.363094" y="149.02231" font-family="sans-serif" font-size="3.5278px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">/dev/spdk/nvme0</tspan></text>
+  <text x="47.625" y="149.02231" fill="#000000" font-family="sans-serif" font-size="10.583px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px" style="line-height:1.25" xml:space="preserve"><tspan x="47.625" y="149.02231" font-family="sans-serif" font-size="3.5278px" stroke-width=".26458" writing-mode="lr" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">/dev/spdk/nvme0n1</tspan></text>
+  <g stroke="#000">
+   <rect x="12.095" y="35.818" width="71.249" height="88.446" ry="4.3467" fill="none" stroke-width=".5"/>
+   <rect x="133.43" y="33.929" width="62.366" height="76.351" ry="4.7247" fill="none" stroke-width=".5"/>
+   <g fill="#fff" stroke-width=".26458">
+    <rect x="14.174" y="91.57" width="64.256" height="24.568"/>
+    <g fill-opacity=".9798">
+     <rect x="46.302" y="100.64" width="26.62" height="11.061"/>
+    </g>
+   </g>
+   <g transform="translate(-.53932 -.16291)">
+    <path d="m63.878 111.98v32.884" fill="none" marker-end="url(#marker1127)" marker-start="url(#Arrow1Lstart)" stroke-width=".26458px"/>
+    <g stroke-width=".265">
+     <path d="m34.585 115.57v28.726" fill="none" marker-end="url(#Arrow1Lend)" marker-start="url(#marker1219)"/>
+     <rect x="136.26" y="39.031" width="54.996" height="58.586" fill="#fff"/>
+     <rect x="153.84" y="52.26" width="34.018" height="11.906" ry="5.8544" fill="none"/>
+    </g>
+    <path d="m112.45 24.479v137.58" fill="none" stroke-dasharray="1.5874999, 1.5874999" stroke-width=".26458"/>
+   </g>
+   <g fill="#fff" stroke-width=".265">
+    <rect x="89.58" y="54.339" width="38.365" height="8.8824"/>
+   </g>
+  </g>
+  <g font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px">
+   <text x="93.54911" y="59.800339" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="93.54911" y="59.800339" stroke-width=".26458">io_msg queue</tspan></text>
+   <text x="11.906249" y="27.31399" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="11.906249" y="27.31399" stroke-width=".26458">CUSE threads</tspan></text>
+   <text x="165.36458" y="27.502975" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="165.36458" y="27.502975" stroke-width=".26458">SPDK threads</tspan></text>
+  </g>
+  <g stroke="#000">
+   <rect x="17.009" y="47.914" width="29.482" height="13.04" ry="6.5201" fill="#fff" stroke-width=".265"/>
+   <rect x="49.921" y="68.161" width="28.915" height="13.04" ry="6.5201" fill="#fff" stroke-width=".265"/>
+   <g fill="none">
+    <path d="m32.506 61.143v30.427" marker-start="url(#marker7156)" stroke-width=".26458px"/>
+    <path d="m63.689 81.176 0.18899 19.277" marker-start="url(#marker4324)" stroke-width=".265"/>
+    <g stroke-width=".26458px">
+     <path d="m46.113 54.339h43.467" marker-end="url(#marker2028)"/>
+     <path d="m64.284 67.972c0.02768-6.3997-1.3229-5.2917 25.135-5.2917" marker-end="url(#marker2110)"/>
+     <path d="m127.78 56.066h25.135" marker-end="url(#marker2300)"/>
+    </g>
+   </g>
+  </g>
+  <g stroke-width=".26458">
+   <g transform="translate(-.25341)" font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" word-spacing="0px">
+    <text x="138.90625" y="44.889877" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="138.90625" y="44.889877" stroke-width=".26458">NVMe</tspan></text>
+    <text x="16.063986" y="97.050598" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="16.063986" y="97.050598" stroke-width=".26458">CUSE ctrlr</tspan></text>
+    <text x="48.380947" y="106.12202" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="48.380947" y="106.12202" stroke-width=".26458">CUSE ns</tspan></text>
+    <text x="51.420551" y="75.799461" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="51.420551" y="75.799461" stroke-width=".26458">ioctl pthread</tspan></text>
+    <text x="18.906757" y="55.833015" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="18.906757" y="55.833015" stroke-width=".26458">ioctl pthread</tspan></text>
+   </g>
+   <path d="m160.86 85.17c0.38097 13.154-7.1538 11.542-82.052 10.936" fill="none" marker-end="url(#marker4572)" stroke="#000" stroke-dasharray="0.79374995, 0.79374995"/>
+   <path d="m179.38 85.17c0.37797 22.25-6.5765 20.83-106.08 20.641" fill="none" marker-end="url(#marker4436)" stroke="#000" stroke-dasharray="0.79374995, 0.79374995"/>
+  </g>
+  <g font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px">
+   <text x="13.229166" y="139.7619" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="13.229166" y="139.7619" stroke-width=".26458">Kernel</tspan></text>
+   <text x="14.552083" y="41.488094" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="14.552083" y="41.488094" stroke-width=".26458">CUSE</tspan></text>
+   <text x="161.73709" y="59.415913" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="161.73709" y="59.415913" stroke-width=".26458">io poller</tspan></text>
+  </g>
+  <g fill="none" stroke="#000">
+   <path d="m111.91 127.5h-109.8" stroke-dasharray="1.58749992, 1.58749992" stroke-width=".26458"/>
+   <rect x="153.3" y="71.941" width="34.018" height="13.229" ry="6.6146" stroke-width=".265"/>
+   <path d="m170.12 64.003v7.9375" marker-end="url(#marker9353)" stroke-width=".265"/>
+  </g>
+  <g font-family="sans-serif" font-size="4.2333px" letter-spacing="0px" stroke-width=".26458" word-spacing="0px">
+   <text x="159.72221" y="79.76664" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="159.72221" y="79.76664" stroke-width=".26458">io execute</tspan></text>
+   <text x="172.34003" y="68.59539" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="172.34003" y="68.59539" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">fn(arg)</tspan></text>
+   <text x="53.046707" y="52.192699" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="53.046707" y="52.192699" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">nvme_io_msg send()</tspan></text>
+   <text x="53.102341" y="60.250244" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="53.102341" y="60.250244" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">nvme_io_msg send()</tspan></text>
+   <text x="120.79763" y="50.70586" font-size="12px" stroke-width="1" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal;line-height:1.25" xml:space="preserve"><tspan x="120.79763" y="50.70586" font-family="sans-serif" font-size="2.8222px" stroke-width=".26458" style="font-feature-settings:normal;font-variant-caps:normal;font-variant-ligatures:normal;font-variant-numeric:normal">spdk_nvme_io_msg process()</tspan></text>
+  </g>
+ </g>
+</svg>

doc/jsonrpc.md

@@ -1627,6 +1627,80 @@ Example response:
 }
 ~~~
 
+## bdev_nvme_cuse_register {#rpc_bdev_nvme_cuse_register}
+
+Register CUSE device on NVMe controller.
+This feature is considered as experimental.
+
+### Parameters
+
+Name                    | Optional | Type        | Description
+----------------------- | -------- | ----------- | -----------
+name                    | Required | string      | Name of the NVMe controller
+dev_path                | Required | string      | Path to the CUSE controller device, e.g. spdk/nvme0
+
+### Example
+
+Example request:
+
+~~~
+{
+  "params": {
+    "dev_path": "spdk/nvme0",
+    "name": "Nvme0"
+  },
+  "jsonrpc": "2.0",
+  "method": "bdev_nvme_cuse_register",
+  "id": 1
+}
+~~~
+
+Example response:
+
+~~~
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "result": true
+}
+~~~
+
+## bdev_nvme_cuse_unregister {#rpc_bdev_nvme_cuse_unregister}
+
+Unregister CUSE device on NVMe controller.
+This feature is considered as experimental.
+
+### Parameters
+
+Name                    | Optional | Type        | Description
+----------------------- | -------- | ----------- | -----------
+name                    | Required | string      | Name of the NVMe controller
+
+### Example
+
+Example request:
+
+~~~
+{
+  "params": {
+    "name": "Nvme0"
+  },
+  "jsonrpc": "2.0",
+  "method": "bdev_nvme_cuse_unregister",
+  "id": 1
+}
+~~~
+
+Example response:
+
+~~~
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "result": true
+}
+~~~
+
 ## bdev_rbd_create {#rpc_bdev_rbd_create}
 
 Create @ref bdev_config_rbd bdev

doc/nvme.md

@@ -9,6 +9,7 @@
 * @ref nvme_fabrics_host
 * @ref nvme_multi_process
 * @ref nvme_hotplug
+* @ref nvme_cuse
 
 # Introduction {#nvme_intro}
 
@@ -266,3 +267,32 @@ This means I/O in flight during a hot remove will complete with an appropriate e
 code and will not crash the application.
 
 @sa spdk_nvme_probe
+
+# NVMe Character Devices {#nvme_cuse}
+
+This feature is considered as experimental.
+
+![NVMe character devices processing diagram](nvme_cuse.svg)
+
+For each controller as well as namespace, character devices are created in the
+locations:
+~~~{.sh}
+    /dev/'dev_path'
+    /dev/'dev_path'nY
+    ...
+~~~
+
+Requests from CUSE are handled by pthreads when controller and namespaces are created.
+During the handling of CUSE requests, operations may be forwarded to an internal NVMe
+queue pair. The user must poll this internal queue pair periodically by polling
+the admin qpair for the associated NVMe controller.
+
+Ioctls that request information attained when attaching NVMe controller receive an
+immediate response, without passing them to the internal queue pair.
+
+This interface reserves one qpair for sending down the I/O for each controller.
+
+## Enabling cuse support for NVMe
+
+Cuse support is disabled by default. To enable support for NVMe devices SPDK
+must be compiled with "./configure --with-nvme-cuse".

dpdk

@@ -1 +1 @@
-Subproject commit 62a2c4c66cd624fd169b7e3148513df306c2dc9d
+Subproject commit 0698cc38e0cb86bc01d82b6f1aef85fd983b213d

include/spdk/nvme.h

@@ -1091,6 +1091,8 @@ struct spdk_nvme_qpair *spdk_nvme_ctrlr_alloc_io_qpair(struct spdk_nvme_ctrlr *c
  * This function is intended to be called on qpairs that have already been connected,
  * but have since entered a failed state as indicated by a return value of -ENXIO from
  * either spdk_nvme_qpair_process_completions or one of the spdk_nvme_ns_cmd_* functions.
+ * This function must be called from the same thread as spdk_nvme_qpair_process_completions
+ * and the spdk_nvme_ns_cmd_* functions.
  *
  * \param qpair The qpair to reconnect.
  *
@@ -1244,26 +1246,6 @@ int spdk_nvme_ctrlr_cmd_io_raw_with_md(struct spdk_nvme_ctrlr *ctrlr,
 int32_t spdk_nvme_qpair_process_completions(struct spdk_nvme_qpair *qpair,
 		uint32_t max_completions);
 
-/**
- * Process IO message sent to controller from external module.
- *
- * This call process requests from the ring, send IO to an allocated qpair or
- * admin commands in its context. This call is non-blocking and intended to be
- * polled by SPDK thread to provide safe environment for NVMe request
- * completition sent by external module to controller.
- *
- * The caller must ensure that each controller is polled by only one thread at
- * a time.
- *
- * This function may be called at any point while the controller is attached to
- * the SPDK NVMe driver.
- *
- * \param ctrlr Opaque handle to NVMe controller.
- *
- * \return number of processed external IO messages.
- */
-int spdk_nvme_io_msg_process(struct spdk_nvme_ctrlr *ctrlr);
-
 /**
  * Send the given admin command to the NVMe controller.
  *
@@ -2603,7 +2585,28 @@ char *spdk_nvme_cuse_get_ctrlr_name(struct spdk_nvme_ctrlr *ctrlr);
  */
 char *spdk_nvme_cuse_get_ns_name(struct spdk_nvme_ctrlr *ctrlr, uint32_t nsid);
 
-int spdk_nvme_cuse_register(struct spdk_nvme_ctrlr *ctrlr, const char *dev_path);
+/**
+ * Create a character device at the path specified (Experimental)
+ *
+ * The character device can handle ioctls and is compatible with a standard
+ * Linux kernel NVMe device. Tools such as nvme-cli can be used to configure
+ * SPDK devices through this interface.
+ *
+ * The user is expected to be polling the admin qpair for this controller periodically
+ * for the CUSE device to function.
+ *
+ * \param ctrlr Opaque handle to the NVMe controller.
+ *
+ * \return 0 on success. Negated errno on failure.
+ */
+int spdk_nvme_cuse_register(struct spdk_nvme_ctrlr *ctrlr);
+
+/**
+ * Remove a previously created character device (Experimental)
+ *
+ * \param ctrlr Opaque handle to the NVMe controller.
+ *
+ */
 void spdk_nvme_cuse_unregister(struct spdk_nvme_ctrlr *ctrlr);
 
 #ifdef __cplusplus

include/spdk/stdinc.h

@@ -57,6 +57,7 @@ extern "C" {
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <strings.h>
 #include <time.h>
 
 /* POSIX */

include/spdk/thread.h

@@ -373,8 +373,12 @@ int spdk_thread_get_stats(struct spdk_thread_stats *stats);
  * \param thread The target thread.
  * \param fn This function will be called on the given thread.
  * \param ctx This context will be passed to fn when called.
+ *
+ * \return 0 on success
+ * \return -ENOMEM if the message could not be allocated
+ * \return -EIO if the message could not be sent to the destination thread
  */
-void spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx);
+int spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx);
 
 /**
  * Send a message to each thread, serially.

include/spdk/version.h

@@ -54,12 +54,12 @@
  * Patch level is incremented on maintenance branch releases and reset to 0 for each
  * new major.minor release.
  */
-#define SPDK_VERSION_PATCH	0
+#define SPDK_VERSION_PATCH	1
 
 /**
  * Version string suffix.
  */
-#define SPDK_VERSION_SUFFIX	"-pre"
+#define SPDK_VERSION_SUFFIX	""
 
 /**
  * Single numeric value representing a version number for compile-time comparisons.

lib/blob/blobstore.c

@@ -698,10 +698,12 @@ _spdk_blob_serialize_extent_rle(const struct spdk_blob *blob,
 	lba_count = lba_per_cluster;
 	extent_idx = 0;
 	for (i = start_cluster + 1; i < blob->active.num_clusters; i++) {
-		if ((lba + lba_count) == blob->active.clusters[i]) {
+		if ((lba + lba_count) == blob->active.clusters[i] && lba != 0) {
+			/* Run-length encode sequential non-zero LBA */
 			lba_count += lba_per_cluster;
 			continue;
 		} else if (lba == 0 && blob->active.clusters[i] == 0) {
+			/* Run-length encode unallocated clusters */
 			lba_count += lba_per_cluster;
 			continue;
 		}

lib/env_dpdk/init.c

@@ -175,6 +175,75 @@ spdk_push_arg(char *args[], int *argcount, char *arg)
 	return tmp;
 }
 
+#if defined(__linux__) && defined(__x86_64__)
+
+/* TODO: Can likely get this value from rlimits in the future */
+#define SPDK_IOMMU_VA_REQUIRED_WIDTH 48
+#define VTD_CAP_MGAW_SHIFT 16
+#define VTD_CAP_MGAW_MASK (0x3F << VTD_CAP_MGAW_SHIFT)
+
+static int
+spdk_get_iommu_width(void)
+{
+	DIR *dir;
+	FILE *file;
+	struct dirent *entry;
+	char mgaw_path[64];
+	char buf[64];
+	char *end;
+	long long int val;
+	int width, tmp;
+
+	dir = opendir("/sys/devices/virtual/iommu/");
+	if (dir == NULL) {
+		return -EINVAL;
+	}
+
+	width = 0;
+
+	while ((entry = readdir(dir)) != NULL) {
+		/* Find directories named "dmar0", "dmar1", etc */
+		if (strncmp(entry->d_name, "dmar", sizeof("dmar") - 1) != 0) {
+			continue;
+		}
+
+		tmp = snprintf(mgaw_path, sizeof(mgaw_path), "/sys/devices/virtual/iommu/%s/intel-iommu/cap",
+			       entry->d_name);
+		if ((unsigned)tmp >= sizeof(mgaw_path)) {
+			continue;
+		}
+
+		file = fopen(mgaw_path, "r");
+		if (file == NULL) {
+			continue;
+		}
+
+		if (fgets(buf, sizeof(buf), file) == NULL) {
+			fclose(file);
+			continue;
+		}
+
+		val = strtoll(buf, &end, 16);
+		if (val == LLONG_MIN || val == LLONG_MAX) {
+			fclose(file);
+			continue;
+		}
+
+		tmp = ((val & VTD_CAP_MGAW_MASK) >> VTD_CAP_MGAW_SHIFT) + 1;
+		if (width == 0 || tmp < width) {
+			width = tmp;
+		}
+
+		fclose(file);
+	}
+
+	closedir(dir);
+
+	return width;
+}
+
+#endif
+
 static int
 spdk_build_eal_cmdline(const struct spdk_env_opts *opts)
 {
@@ -336,6 +405,29 @@ spdk_build_eal_cmdline(const struct spdk_env_opts *opts)
 	}
 
 #ifdef __linux__
+
+#if defined(__x86_64__)
+	/* DPDK by default guesses that it should be using iova-mode=va so that it can
+	 * support running as an unprivileged user. However, some systems (especially
+	 * virtual machines) don't have an IOMMU capable of handling the full virtual
+	 * address space and DPDK doesn't currently catch that. Add a check in SPDK
+	 * and force iova-mode=pa here. */
+	if (spdk_get_iommu_width() < SPDK_IOMMU_VA_REQUIRED_WIDTH) {
+		args = spdk_push_arg(args, &argcount, _sprintf_alloc("--iova-mode=pa"));
+		if (args == NULL) {
+			return -1;
+		}
+	}
+#elif defined(__PPC64__)
+	/* On Linux + PowerPC, DPDK doesn't support VA mode at all. Unfortunately, it doesn't correctly
+	 * auto-detect at the moment, so we'll just force it here. */
+	args = spdk_push_arg(args, &argcount, _sprintf_alloc("--iova-mode=pa"));
+	if (args == NULL) {
+		return -1;
+	}
+#endif
+
+
 	/* Set the base virtual address - it must be an address that is not in the
 	 * ASAN shadow region, otherwise ASAN-enabled builds will ignore the
 	 * mmap hint.

lib/iscsi/conn.c

@@ -377,12 +377,6 @@ _iscsi_conn_free(struct spdk_iscsi_conn *conn)
 
 	spdk_iscsi_param_free(conn->params);
 
-	/*
-	 * Each connection pre-allocates its next PDU - make sure these get
-	 *  freed here.
-	 */
-	spdk_put_pdu(conn->pdu_in_progress);
-
 	free_conn(conn);
 }
 
@@ -675,6 +669,10 @@ _iscsi_conn_check_pending_tasks(void *arg)
 void
 spdk_iscsi_conn_destruct(struct spdk_iscsi_conn *conn)
 {
+	struct spdk_iscsi_pdu *pdu;
+	struct spdk_iscsi_task *task;
+	int opcode;
+
 	/* If a connection is already in exited status, just return */
 	if (conn->state >= ISCSI_CONN_STATE_EXITED) {
 		return;
@@ -682,6 +680,32 @@ spdk_iscsi_conn_destruct(struct spdk_iscsi_conn *conn)
 
 	conn->state = ISCSI_CONN_STATE_EXITED;
 
+	/*
+	 * Each connection pre-allocates its next PDU - make sure these get
+	 *  freed here.
+	 */
+	pdu = conn->pdu_in_progress;
+	if (pdu) {
+		/* remove the task left in the PDU too. */
+		task = pdu->task;
+		if (task) {
+			opcode = pdu->bhs.opcode;
+			switch (opcode) {
+			case ISCSI_OP_SCSI:
+			case ISCSI_OP_SCSI_DATAOUT:
+				spdk_scsi_task_process_abort(&task->scsi);
+				spdk_iscsi_task_cpl(&task->scsi);
+				break;
+			default:
+				SPDK_ERRLOG("unexpected opcode %x\n", opcode);
+				spdk_iscsi_task_put(task);
+				break;
+			}
+		}
+		spdk_put_pdu(pdu);
+		conn->pdu_in_progress = NULL;
+	}
+
 	if (conn->sess != NULL && conn->pending_task_cnt > 0) {
 		iscsi_conn_cleanup_backend(conn);
 	}

lib/iscsi/iscsi.c

@@ -4385,8 +4385,11 @@ iscsi_pdu_hdr_op_data(struct spdk_iscsi_conn *conn, struct spdk_iscsi_pdu *pdu)
 	struct spdk_scsi_lun	*lun_dev;
 	uint32_t transfer_tag;
 	uint32_t task_tag;
+	uint32_t transfer_len;
 	uint32_t DataSN;
 	uint32_t buffer_offset;
+	uint32_t len;
+	int F_bit;
 	int rc;
 	int reject_reason = ISCSI_REASON_INVALID_PDU_FIELD;
 
@@ -4396,6 +4399,7 @@ iscsi_pdu_hdr_op_data(struct spdk_iscsi_conn *conn, struct spdk_iscsi_pdu *pdu)
 	}
 
 	reqh = (struct iscsi_bhs_data_out *)&pdu->bhs;
+	F_bit = !!(reqh->flags & ISCSI_FLAG_FINAL);
 	transfer_tag = from_be32(&reqh->ttt);
 	task_tag = from_be32(&reqh->itt);
 	DataSN = from_be32(&reqh->data_sn);
@@ -4440,13 +4444,26 @@ iscsi_pdu_hdr_op_data(struct spdk_iscsi_conn *conn, struct spdk_iscsi_pdu *pdu)
 		return SPDK_ISCSI_CONNECTION_FATAL;
 	}
 
-	if (task->current_r2t_length + pdu->data_segment_len > conn->sess->MaxBurstLength) {
-		SPDK_ERRLOG("R2T burst(%zu) > MaxBurstLength(%u)\n",
-			    task->current_r2t_length + pdu->data_segment_len,
+	transfer_len = task->scsi.transfer_len;
+	task->current_r2t_length += pdu->data_segment_len;
+	task->next_expected_r2t_offset += pdu->data_segment_len;
+	task->r2t_datasn++;
+
+	if (task->current_r2t_length > conn->sess->MaxBurstLength) {
+		SPDK_ERRLOG("R2T burst(%u) > MaxBurstLength(%u)\n",
+			    task->current_r2t_length,
 			    conn->sess->MaxBurstLength);
 		return SPDK_ISCSI_CONNECTION_FATAL;
 	}
 
+	if (F_bit) {
+		/*
+		 * This R2T burst is done. Clear the length before we
+		 *  receive a PDU for the next R2t burst.
+		 */
+		task->current_r2t_length = 0;
+	}
+
 	subtask = spdk_iscsi_task_get(conn, task, spdk_iscsi_task_cpl);
 	if (subtask == NULL) {
 		SPDK_ERRLOG("Unable to acquire subtask\n");
@@ -4456,6 +4473,20 @@ iscsi_pdu_hdr_op_data(struct spdk_iscsi_conn *conn, struct spdk_iscsi_pdu *pdu)
 	subtask->scsi.length = pdu->data_segment_len;
 	spdk_iscsi_task_associate_pdu(subtask, pdu);
 
+	if (task->next_expected_r2t_offset == transfer_len) {
+		task->acked_r2tsn++;
+	} else if (F_bit && (task->next_r2t_offset < transfer_len)) {
+		task->acked_r2tsn++;
+		len = DMIN32(conn->sess->MaxBurstLength, (transfer_len -
+				task->next_r2t_offset));
+		rc = iscsi_send_r2t(conn, task, task->next_r2t_offset, len,
+				    task->ttt, &task->R2TSN);
+		if (rc < 0) {
+			SPDK_ERRLOG("iscsi_send_r2t() failed\n");
+		}
+		task->next_r2t_offset += len;
+	}
+
 	if (lun_dev == NULL) {
 		SPDK_DEBUGLOG(SPDK_LOG_ISCSI, "LUN %d is removed, complete the task immediately\n",
 			      task->lun_id);
@@ -4481,36 +4512,13 @@ reject_return:
 static int
 iscsi_pdu_payload_op_data(struct spdk_iscsi_conn *conn, struct spdk_iscsi_pdu *pdu)
 {
-	struct spdk_iscsi_task	*task, *subtask;
-	struct iscsi_bhs_data_out *reqh;
-	uint32_t transfer_len;
-	uint32_t len;
-	int F_bit;
-	int rc;
+	struct spdk_iscsi_task	*subtask;
 
 	if (pdu->task == NULL) {
 		return 0;
 	}
 
 	subtask = pdu->task;
-	task = spdk_iscsi_task_get_primary(subtask);
-	assert(task != subtask);
-
-	reqh = (struct iscsi_bhs_data_out *)&pdu->bhs;
-	F_bit = !!(reqh->flags & ISCSI_FLAG_FINAL);
-
-	transfer_len = task->scsi.transfer_len;
-	task->current_r2t_length += pdu->data_segment_len;
-	task->next_expected_r2t_offset += pdu->data_segment_len;
-	task->r2t_datasn++;
-
-	if (F_bit) {
-		/*
-		 * This R2T burst is done.  Clear the length before we
-		 *  receive a PDU for the next R2T burst.
-		 */
-		task->current_r2t_length = 0;
-	}
 
 	if (spdk_likely(!pdu->dif_insert_or_strip)) {
 		spdk_scsi_task_set_data(&subtask->scsi, pdu->data, pdu->data_segment_len);
@@ -4518,23 +4526,9 @@ iscsi_pdu_payload_op_data(struct spdk_iscsi_conn *conn, struct spdk_iscsi_pdu *p
 		spdk_scsi_task_set_data(&subtask->scsi, pdu->data, pdu->data_buf_len);
 	}
 
-	if (task->next_expected_r2t_offset == transfer_len) {
-		task->acked_r2tsn++;
-	} else if (F_bit && (task->next_r2t_offset < transfer_len)) {
-		task->acked_r2tsn++;
-		len = DMIN32(conn->sess->MaxBurstLength, (transfer_len -
-				task->next_r2t_offset));
-		rc = iscsi_send_r2t(conn, task, task->next_r2t_offset, len,
-				    task->ttt, &task->R2TSN);
-		if (rc < 0) {
-			SPDK_ERRLOG("iscsi_send_r2t() failed\n");
-		}
-		task->next_r2t_offset += len;
-	}
-
-	if (spdk_scsi_dev_get_lun(conn->dev, task->lun_id) == NULL) {
+	if (spdk_scsi_dev_get_lun(conn->dev, subtask->lun_id) == NULL) {
 		SPDK_DEBUGLOG(SPDK_LOG_ISCSI, "LUN %d is removed, complete the task immediately\n",
-			      task->lun_id);
+			      subtask->lun_id);
 		subtask->scsi.transfer_len = subtask->scsi.length;
 		spdk_scsi_task_process_null_lun(&subtask->scsi);
 		spdk_iscsi_task_cpl(&subtask->scsi);
@@ -4976,8 +4970,6 @@ iscsi_read_pdu(struct spdk_iscsi_conn *conn)
 			}
 			break;
 		case ISCSI_PDU_RECV_STATE_ERROR:
-			spdk_put_pdu(pdu);
-			conn->pdu_in_progress = NULL;
 			return SPDK_ISCSI_CONNECTION_FATAL;
 		default:
 			assert(false);

lib/jsonrpc/jsonrpc_client.c

@@ -93,7 +93,7 @@ spdk_jsonrpc_parse_response(struct spdk_jsonrpc_client *client)
 	}
 
 	SPDK_DEBUGLOG(SPDK_LOG_RPC_CLIENT, "JSON string is :\n%s\n", client->recv_buf);
-	if (rc < 0 || rc > SPDK_JSONRPC_MAX_VALUES) {
+	if (rc < 0 || rc > SPDK_JSONRPC_CLIENT_MAX_VALUES) {
 		SPDK_ERRLOG("JSON parse error (rc: %zd)\n", rc);
 		/*
 		 * Can't recover from parse error (no guaranteed resync point in streaming JSON).

lib/jsonrpc/jsonrpc_internal.h

@@ -46,6 +46,7 @@
 #define SPDK_JSONRPC_ID_MAX_LEN		128
 #define SPDK_JSONRPC_MAX_CONNS		64
 #define SPDK_JSONRPC_MAX_VALUES		1024
+#define SPDK_JSONRPC_CLIENT_MAX_VALUES		8192
 
 struct spdk_jsonrpc_request {
 	struct spdk_jsonrpc_server_conn *conn;

lib/nvme/nvme.c

@@ -69,7 +69,7 @@ spdk_nvme_detach(struct spdk_nvme_ctrlr *ctrlr)
 	nvme_ctrlr_proc_put_ref(ctrlr);
 
 	if (nvme_ctrlr_get_ref_count(ctrlr) == 0) {
-		nvme_io_msg_ctrlr_stop(ctrlr, NULL, true);
+		nvme_io_msg_ctrlr_detach(ctrlr);
 		if (nvme_ctrlr_shared(ctrlr)) {
 			TAILQ_REMOVE(&g_spdk_nvme_driver->shared_attached_ctrlrs, ctrlr, tailq);
 		} else {

lib/nvme/nvme_ctrlr.c

@@ -34,6 +34,7 @@
 #include "spdk/stdinc.h"
 
 #include "nvme_internal.h"
+#include "nvme_io_msg.h"
 
 #include "spdk/env.h"
 #include "spdk/string.h"
@@ -412,19 +413,42 @@ spdk_nvme_ctrlr_reconnect_io_qpair(struct spdk_nvme_qpair *qpair)
 		goto out;
 	}
 
+	/* We have to confirm that any old memory is cleaned up. */
+	nvme_transport_ctrlr_disconnect_qpair(ctrlr, qpair);
+
 	rc = nvme_transport_ctrlr_connect_qpair(ctrlr, qpair);
 	if (rc) {
 		nvme_qpair_set_state(qpair, NVME_QPAIR_DISABLED);
+		qpair->transport_qp_is_failed = true;
 		rc = -EAGAIN;
 		goto out;
 	}
 	nvme_qpair_set_state(qpair, NVME_QPAIR_CONNECTED);
+	qpair->transport_qp_is_failed = false;
 
 out:
 	nvme_robust_mutex_unlock(&ctrlr->ctrlr_lock);
 	return rc;
 }
 
+/*
+ * This internal function will attempt to take the controller
+ * lock before calling disconnect on a controller qpair.
+ * Functions already holding the controller lock should
+ * call nvme_transport_ctrlr_disconnect_qpair directly.
+ */
+void
+nvme_ctrlr_disconnect_qpair(struct spdk_nvme_qpair *qpair)
+{
+	struct spdk_nvme_ctrlr *ctrlr = qpair->ctrlr;
+
+	assert(ctrlr != NULL);
+
+	nvme_robust_mutex_lock(&ctrlr->ctrlr_lock);
+	nvme_transport_ctrlr_disconnect_qpair(ctrlr, qpair);
+	nvme_robust_mutex_unlock(&ctrlr->ctrlr_lock);
+}
+
 int
 spdk_nvme_ctrlr_free_io_qpair(struct spdk_nvme_qpair *qpair)
 {
@@ -1055,7 +1079,7 @@ spdk_nvme_ctrlr_reset(struct spdk_nvme_ctrlr *ctrlr)
 	/* Disable all queues before disabling the controller hardware. */
 	TAILQ_FOREACH(qpair, &ctrlr->active_io_qpairs, tailq) {
 		nvme_qpair_set_state(qpair, NVME_QPAIR_DISABLED);
-		nvme_transport_ctrlr_disconnect_qpair(ctrlr, qpair);
+		qpair->transport_qp_is_failed = true;
 	}
 	nvme_qpair_set_state(ctrlr->adminq, NVME_QPAIR_DISABLED);
 	nvme_qpair_complete_error_reqs(ctrlr->adminq);
@@ -1084,7 +1108,14 @@ spdk_nvme_ctrlr_reset(struct spdk_nvme_ctrlr *ctrlr)
 		}
 	}
 
-	if (rc == 0) {
+	/*
+	 * For PCIe controllers, the memory locations of the tranpsort qpair
+	 * don't change when the controller is reset. They simply need to be
+	 * re-enabled with admin commands to the controller. For fabric
+	 * controllers we need to disconnect and reconnect the qpair on its
+	 * own thread outside of the context of the reset.
+	 */
+	if (rc == 0 && ctrlr->trid.trtype == SPDK_NVME_TRANSPORT_PCIE) {
 		/* Reinitialize qpairs */
 		TAILQ_FOREACH(qpair, &ctrlr->active_io_qpairs, tailq) {
 			if (nvme_transport_ctrlr_connect_qpair(ctrlr, qpair) != 0) {
@@ -1093,6 +1124,7 @@ spdk_nvme_ctrlr_reset(struct spdk_nvme_ctrlr *ctrlr)
 				continue;
 			}
 			nvme_qpair_set_state(qpair, NVME_QPAIR_CONNECTED);
+			qpair->transport_qp_is_failed = false;
 		}
 	}
 
@@ -2615,14 +2647,30 @@ int32_t
 spdk_nvme_ctrlr_process_admin_completions(struct spdk_nvme_ctrlr *ctrlr)
 {
 	int32_t num_completions;
+	int32_t rc;
 
 	nvme_robust_mutex_lock(&ctrlr->ctrlr_lock);
+
 	if (ctrlr->keep_alive_interval_ticks) {
 		nvme_ctrlr_keep_alive(ctrlr);
 	}
-	num_completions = spdk_nvme_qpair_process_completions(ctrlr->adminq, 0);
+
+	rc = spdk_nvme_io_msg_process(ctrlr);
+	if (rc < 0) {
+		nvme_robust_mutex_unlock(&ctrlr->ctrlr_lock);
+		return rc;
+	}
+	num_completions = rc;
+
+	rc = spdk_nvme_qpair_process_completions(ctrlr->adminq, 0);
 	nvme_robust_mutex_unlock(&ctrlr->ctrlr_lock);
 
+	if (rc < 0) {
+		num_completions = rc;
+	} else {
+		num_completions += rc;
+	}
+
 	return num_completions;
 }
 

lib/nvme/nvme_cuse.c

@@ -44,6 +44,9 @@
 
 struct cuse_device {
 	char				dev_name[128];
+	uint32_t			index;
+	int				claim_fd;
+	char				lock_name[64];
 
 	struct spdk_nvme_ctrlr		*ctrlr;		/**< NVMe controller */
 	uint32_t			nsid;		/**< NVMe name space id, or 0 */
@@ -58,6 +61,7 @@ struct cuse_device {
 };
 
 static TAILQ_HEAD(, cuse_device) g_ctrlr_ctx_head = TAILQ_HEAD_INITIALIZER(g_ctrlr_ctx_head);
+static struct spdk_bit_array *g_ctrlr_started;
 
 struct cuse_io_ctx {
 	struct spdk_nvme_cmd	nvme_cmd;
@@ -623,6 +627,10 @@ cuse_thread(void *arg)
 	const char *dev_info_argv[] = { devname_arg };
 	struct cuse_info ci;
 	int multithreaded;
+	int rc;
+	struct fuse_buf buf = { .mem = NULL };
+	struct pollfd fds;
+	int timeout_msecs = 500;
 
 	spdk_unaffinitize_thread();
 
@@ -646,7 +654,22 @@ cuse_thread(void *arg)
 	}
 
 	SPDK_NOTICELOG("fuse session for device %s created\n", cuse_device->dev_name);
-	fuse_session_loop(cuse_device->session);
+
+	/* Receive and process fuse requests */
+	fds.fd = fuse_session_fd(cuse_device->session);
+	fds.events = POLLIN;
+	while (!fuse_session_exited(cuse_device->session)) {
+		rc = poll(&fds, 1, timeout_msecs);
+		if (rc <= 0) {
+			continue;
+		}
+		rc = fuse_session_receive_buf(cuse_device->session, &buf);
+		if (rc > 0) {
+			fuse_session_process_buf(cuse_device->session, &buf);
+		}
+	}
+	free(buf.mem);
+	fuse_session_reset(cuse_device->session);
 
 end:
 	cuse_lowlevel_teardown(cuse_device->session);
@@ -661,6 +684,7 @@ static int
 cuse_nvme_ns_start(struct cuse_device *ctrlr_device, uint32_t nsid, const char *dev_path)
 {
 	struct cuse_device *ns_device;
+	int rv;
 
 	ns_device = (struct cuse_device *)calloc(1, sizeof(struct cuse_device));
 	if (!ns_device) {
@@ -671,8 +695,13 @@ cuse_nvme_ns_start(struct cuse_device *ctrlr_device, uint32_t nsid, const char *
 	ns_device->ctrlr = ctrlr_device->ctrlr;
 	ns_device->ctrlr_device = ctrlr_device;
 	ns_device->nsid = nsid;
-	snprintf(ns_device->dev_name, sizeof(ns_device->dev_name), "%sn%d",
-		 dev_path, ns_device->nsid);
+	rv = snprintf(ns_device->dev_name, sizeof(ns_device->dev_name), "%sn%d",
+		      dev_path, ns_device->nsid);
+	if (rv < 0) {
+		SPDK_ERRLOG("Device name too long.\n");
+		free(ns_device);
+		return -1;
+	}
 
 	if (pthread_create(&ns_device->tid, NULL, cuse_thread, ns_device)) {
 		SPDK_ERRLOG("pthread_create failed\n");
@@ -684,6 +713,68 @@ cuse_nvme_ns_start(struct cuse_device *ctrlr_device, uint32_t nsid, const char *
 	return 0;
 }
 
+static int
+nvme_cuse_claim(struct cuse_device *ctrlr_device, uint32_t index)
+{
+	int dev_fd;
+	int pid;
+	void *dev_map;
+	struct flock cusedev_lock = {
+		.l_type = F_WRLCK,
+		.l_whence = SEEK_SET,
+		.l_start = 0,
+		.l_len = 0,
+	};
+
+	snprintf(ctrlr_device->lock_name, sizeof(ctrlr_device->lock_name),
+		 "/tmp/spdk_nvme_cuse_lock_%" PRIu32, index);
+
+	dev_fd = open(ctrlr_device->lock_name, O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
+	if (dev_fd == -1) {
+		fprintf(stderr, "could not open %s\n", ctrlr_device->lock_name);
+		return -errno;
+	}
+
+	if (ftruncate(dev_fd, sizeof(int)) != 0) {
+		fprintf(stderr, "could not truncate %s\n", ctrlr_device->lock_name);
+		close(dev_fd);
+		return -errno;
+	}
+
+	dev_map = mmap(NULL, sizeof(int), PROT_READ | PROT_WRITE,
+		       MAP_SHARED, dev_fd, 0);
+	if (dev_map == MAP_FAILED) {
+		fprintf(stderr, "could not mmap dev %s (%d)\n", ctrlr_device->lock_name, errno);
+		close(dev_fd);
+		return -errno;
+	}
+
+	if (fcntl(dev_fd, F_SETLK, &cusedev_lock) != 0) {
+		pid = *(int *)dev_map;
+		fprintf(stderr, "Cannot create lock on device %s, probably"
+			" process %d has claimed it\n", ctrlr_device->lock_name, pid);
+		munmap(dev_map, sizeof(int));
+		close(dev_fd);
+		/* F_SETLK returns unspecified errnos, normalize them */
+		return -EACCES;
+	}
+
+	*(int *)dev_map = (int)getpid();
+	munmap(dev_map, sizeof(int));
+	ctrlr_device->claim_fd = dev_fd;
+	ctrlr_device->index = index;
+	/* Keep dev_fd open to maintain the lock. */
+	return 0;
+}
+
+static void
+nvme_cuse_unclaim(struct cuse_device *ctrlr_device)
+{
+	close(ctrlr_device->claim_fd);
+	ctrlr_device->claim_fd = -1;
+	unlink(ctrlr_device->lock_name);
+}
+
 static void
 cuse_nvme_ctrlr_stop(struct cuse_device *ctrlr_device)
 {
@@ -691,41 +782,71 @@ cuse_nvme_ctrlr_stop(struct cuse_device *ctrlr_device)
 
 	TAILQ_FOREACH_SAFE(ns_device, &ctrlr_device->ns_devices, tailq, tmp) {
 		fuse_session_exit(ns_device->session);
-		pthread_kill(ns_device->tid, SIGHUP);
 		pthread_join(ns_device->tid, NULL);
 		TAILQ_REMOVE(&ctrlr_device->ns_devices, ns_device, tailq);
 		free(ns_device);
 	}
 
 	fuse_session_exit(ctrlr_device->session);
-	pthread_kill(ctrlr_device->tid, SIGHUP);
 	pthread_join(ctrlr_device->tid, NULL);
 	TAILQ_REMOVE(&g_ctrlr_ctx_head, ctrlr_device, tailq);
+	spdk_bit_array_clear(g_ctrlr_started, ctrlr_device->index);
+	if (spdk_bit_array_count_set(g_ctrlr_started) == 0) {
+		spdk_bit_array_free(&g_ctrlr_started);
+	}
+	nvme_cuse_unclaim(ctrlr_device);
 	free(ctrlr_device);
 }
 
 static int
-nvme_cuse_start(struct spdk_nvme_ctrlr *ctrlr, const char *dev_path)
+nvme_cuse_start(struct spdk_nvme_ctrlr *ctrlr)
 {
 	uint32_t i, nsid;
+	int rv = 0;
 	struct cuse_device *ctrlr_device;
 
 	SPDK_NOTICELOG("Creating cuse device for controller\n");
 
+	if (g_ctrlr_started == NULL) {
+		g_ctrlr_started = spdk_bit_array_create(128);
+		if (g_ctrlr_started == NULL) {
+			SPDK_ERRLOG("Cannot create bit array\n");
+			return -1;
+		}
+	}
+
 	ctrlr_device = (struct cuse_device *)calloc(1, sizeof(struct cuse_device));
 	if (!ctrlr_device) {
 		SPDK_ERRLOG("Cannot allocate memory for ctrlr_device.");
-		return -ENOMEM;
+		rv = -ENOMEM;
+		goto err2;
 	}
 
 	TAILQ_INIT(&ctrlr_device->ns_devices);
 	ctrlr_device->ctrlr = ctrlr;
-	snprintf(ctrlr_device->dev_name, sizeof(ctrlr_device->dev_name), "%s", dev_path);
+
+	/* Check if device already exists, if not increment index until success */
+	ctrlr_device->index = 0;
+	while (1) {
+		ctrlr_device->index = spdk_bit_array_find_first_clear(g_ctrlr_started, ctrlr_device->index);
+		if (ctrlr_device->index == UINT32_MAX) {
+			SPDK_ERRLOG("Too many registered controllers\n");
+			goto err2;
+		}
+
+		if (nvme_cuse_claim(ctrlr_device, ctrlr_device->index) == 0) {
+			break;
+		}
+		ctrlr_device->index++;
+	}
+	spdk_bit_array_set(g_ctrlr_started, ctrlr_device->index);
+	snprintf(ctrlr_device->dev_name, sizeof(ctrlr_device->dev_name), "spdk/nvme%d",
+		 ctrlr_device->index);
 
 	if (pthread_create(&ctrlr_device->tid, NULL, cuse_thread, ctrlr_device)) {
 		SPDK_ERRLOG("pthread_create failed\n");
-		free(ctrlr_device);
-		return -1;
+		rv = -1;
+		goto err3;
 	}
 	TAILQ_INSERT_TAIL(&g_ctrlr_ctx_head, ctrlr_device, tailq);
 
@@ -736,14 +857,24 @@ nvme_cuse_start(struct spdk_nvme_ctrlr *ctrlr, const char *dev_path)
 			continue;
 		}
 
-		if (cuse_nvme_ns_start(ctrlr_device, nsid, dev_path) < 0) {
+		if (cuse_nvme_ns_start(ctrlr_device, nsid, ctrlr_device->dev_name) < 0) {
 			SPDK_ERRLOG("Cannot start CUSE namespace device.");
 			cuse_nvme_ctrlr_stop(ctrlr_device);
-			return -1;
+			rv = -1;
+			goto err3;
 		}
 	}
 
 	return 0;
+
+err3:
+	spdk_bit_array_clear(g_ctrlr_started, ctrlr_device->index);
+err2:
+	free(ctrlr_device);
+	if (spdk_bit_array_count_set(g_ctrlr_started) == 0) {
+		spdk_bit_array_free(&g_ctrlr_started);
+	}
+	return rv;
 }
 
 static void
@@ -767,25 +898,22 @@ nvme_cuse_stop(struct spdk_nvme_ctrlr *ctrlr)
 
 static struct nvme_io_msg_producer cuse_nvme_io_msg_producer = {
 	.name = "cuse",
+	.stop = nvme_cuse_stop,
 };
 
 int
-spdk_nvme_cuse_register(struct spdk_nvme_ctrlr *ctrlr, const char *dev_path)
+spdk_nvme_cuse_register(struct spdk_nvme_ctrlr *ctrlr)
 {
 	int rc;
 
-	if (dev_path == NULL) {
-		return -EINVAL;
-	}
-
-	rc = nvme_io_msg_ctrlr_start(ctrlr, &cuse_nvme_io_msg_producer);
+	rc = nvme_io_msg_ctrlr_register(ctrlr, &cuse_nvme_io_msg_producer);
 	if (rc) {
 		return rc;
 	}
 
-	rc = nvme_cuse_start(ctrlr, dev_path);
+	rc = nvme_cuse_start(ctrlr);
 	if (rc) {
-		nvme_io_msg_ctrlr_stop(ctrlr, &cuse_nvme_io_msg_producer, false);
+		nvme_io_msg_ctrlr_unregister(ctrlr, &cuse_nvme_io_msg_producer);
 	}
 
 	return rc;
@@ -796,7 +924,7 @@ spdk_nvme_cuse_unregister(struct spdk_nvme_ctrlr *ctrlr)
 {
 	nvme_cuse_stop(ctrlr);
 
-	nvme_io_msg_ctrlr_stop(ctrlr, &cuse_nvme_io_msg_producer, false);
+	nvme_io_msg_ctrlr_unregister(ctrlr, &cuse_nvme_io_msg_producer);
 }
 
 char *

lib/nvme/nvme_fabric.c

@@ -240,10 +240,10 @@ nvme_fabric_ctrlr_scan(struct spdk_nvme_probe_ctx *probe_ctx,
 	discovery_opts.keep_alive_timeout_ms = 0;
 
 	discovery_ctrlr = nvme_transport_ctrlr_construct(&probe_ctx->trid, &discovery_opts, NULL);
-	nvme_qpair_set_state(discovery_ctrlr->adminq, NVME_QPAIR_ENABLED);
 	if (discovery_ctrlr == NULL) {
 		return -1;
 	}
+	nvme_qpair_set_state(discovery_ctrlr->adminq, NVME_QPAIR_ENABLED);
 
 	/* TODO: this should be using the normal NVMe controller initialization process +1 */
 	cc.raw = 0;

lib/nvme/nvme_internal.h

@@ -858,10 +858,11 @@ int	nvme_ctrlr_get_vs(struct spdk_nvme_ctrlr *ctrlr, union spdk_nvme_vs_register
 int	nvme_ctrlr_get_cmbsz(struct spdk_nvme_ctrlr *ctrlr, union spdk_nvme_cmbsz_register *cmbsz);
 void	nvme_ctrlr_init_cap(struct spdk_nvme_ctrlr *ctrlr, const union spdk_nvme_cap_register *cap,
 			    const union spdk_nvme_vs_register *vs);
-int	nvme_qpair_init(struct spdk_nvme_qpair *qpair, uint16_t id,
-			struct spdk_nvme_ctrlr *ctrlr,
-			enum spdk_nvme_qprio qprio,
-			uint32_t num_requests);
+void nvme_ctrlr_disconnect_qpair(struct spdk_nvme_qpair *qpair);
+int nvme_qpair_init(struct spdk_nvme_qpair *qpair, uint16_t id,
+		    struct spdk_nvme_ctrlr *ctrlr,
+		    enum spdk_nvme_qprio qprio,
+		    uint32_t num_requests);
 void	nvme_qpair_deinit(struct spdk_nvme_qpair *qpair);
 void	nvme_qpair_complete_error_reqs(struct spdk_nvme_qpair *qpair);
 int	nvme_qpair_submit_request(struct spdk_nvme_qpair *qpair,

lib/nvme/nvme_io_msg.c

@@ -107,7 +107,8 @@ spdk_nvme_io_msg_process(struct spdk_nvme_ctrlr *ctrlr)
 }
 
 int
-nvme_io_msg_ctrlr_start(struct spdk_nvme_ctrlr *ctrlr, struct nvme_io_msg_producer *io_msg_producer)
+nvme_io_msg_ctrlr_register(struct spdk_nvme_ctrlr *ctrlr,
+			   struct nvme_io_msg_producer *io_msg_producer)
 {
 	if (io_msg_producer == NULL) {
 		SPDK_ERRLOG("io_msg_producer cannot be NULL\n");
@@ -144,29 +145,37 @@ nvme_io_msg_ctrlr_start(struct spdk_nvme_ctrlr *ctrlr, struct nvme_io_msg_produc
 }
 
 void
-nvme_io_msg_ctrlr_stop(struct spdk_nvme_ctrlr *ctrlr, struct nvme_io_msg_producer *io_msg_producer,
-		       bool shutdown)
+nvme_io_msg_ctrlr_detach(struct spdk_nvme_ctrlr *ctrlr)
 {
-	if (STAILQ_EMPTY(&ctrlr->io_producers) && shutdown) {
-		/* Shutdown path with no producers registered = io msg ctrlr not started */
-		return;
-	}
+	struct nvme_io_msg_producer *io_msg_producer, *tmp;
 
-	if (io_msg_producer != NULL) {
+	/* Stop all producers */
+	STAILQ_FOREACH_SAFE(io_msg_producer, &ctrlr->io_producers, link, tmp) {
+		io_msg_producer->stop(ctrlr);
 		STAILQ_REMOVE(&ctrlr->io_producers, io_msg_producer, nvme_io_msg_producer, link);
 	}
 
-	if (!STAILQ_EMPTY(&ctrlr->io_producers) && !shutdown) {
-		/* There are still some registered producers */
-		return;
+	if (ctrlr->external_io_msgs) {
+		spdk_ring_free(ctrlr->external_io_msgs);
 	}
 
-	assert(ctrlr->external_io_msgs);
-	spdk_ring_free(ctrlr->external_io_msgs);
-
 	if (ctrlr->external_io_msgs_qpair) {
 		spdk_nvme_ctrlr_free_io_qpair(ctrlr->external_io_msgs_qpair);
+		ctrlr->external_io_msgs_qpair = NULL;
 	}
 
 	pthread_mutex_destroy(&ctrlr->external_io_msgs_lock);
 }
+
+void
+nvme_io_msg_ctrlr_unregister(struct spdk_nvme_ctrlr *ctrlr,
+			     struct nvme_io_msg_producer *io_msg_producer)
+{
+	if (io_msg_producer != NULL) {
+		STAILQ_REMOVE(&ctrlr->io_producers, io_msg_producer, nvme_io_msg_producer, link);
+	}
+
+	if (STAILQ_EMPTY(&ctrlr->io_producers)) {
+		nvme_io_msg_ctrlr_detach(ctrlr);
+	}
+}

lib/nvme/nvme_io_msg.h

@@ -52,15 +52,37 @@ struct spdk_nvme_io_msg {
 
 struct nvme_io_msg_producer {
 	const char *name;
+	void (*stop)(struct spdk_nvme_ctrlr *ctrlr);
 	STAILQ_ENTRY(nvme_io_msg_producer) link;
 };
 
 int nvme_io_msg_send(struct spdk_nvme_ctrlr *ctrlr, uint32_t nsid, spdk_nvme_io_msg_fn fn,
 		     void *arg);
 
-int nvme_io_msg_ctrlr_start(struct spdk_nvme_ctrlr *ctrlr,
-			    struct nvme_io_msg_producer *io_msg_producer);
-void nvme_io_msg_ctrlr_stop(struct spdk_nvme_ctrlr *ctrlr,
-			    struct nvme_io_msg_producer *io_msg_producer, bool shutdown);
+/**
+ * Process IO message sent to controller from external module.
+ *
+ * This call process requests from the ring, send IO to an allocated qpair or
+ * admin commands in its context. This call is non-blocking and intended to be
+ * polled by SPDK thread to provide safe environment for NVMe request
+ * completition sent by external module to controller.
+ *
+ * The caller must ensure that each controller is polled by only one thread at
+ * a time.
+ *
+ * This function may be called at any point while the controller is attached to
+ * the SPDK NVMe driver.
+ *
+ * \param ctrlr Opaque handle to NVMe controller.
+ *
+ * \return number of processed external IO messages.
+ */
+int spdk_nvme_io_msg_process(struct spdk_nvme_ctrlr *ctrlr);
+
+int nvme_io_msg_ctrlr_register(struct spdk_nvme_ctrlr *ctrlr,
+			       struct nvme_io_msg_producer *io_msg_producer);
+void nvme_io_msg_ctrlr_unregister(struct spdk_nvme_ctrlr *ctrlr,
+				  struct nvme_io_msg_producer *io_msg_producer);
+void nvme_io_msg_ctrlr_detach(struct spdk_nvme_ctrlr *ctrlr);
 
 #endif /* SPDK_NVME_IO_MSG_H_ */

lib/nvme/nvme_pcie.c

@@ -212,7 +212,7 @@ static int nvme_pcie_qpair_destroy(struct spdk_nvme_qpair *qpair);
 __thread struct nvme_pcie_ctrlr *g_thread_mmio_ctrlr = NULL;
 static uint16_t g_signal_lock;
 static bool g_sigset = false;
-static int hotplug_fd = -1;
+static int g_hotplug_fd = -1;
 
 static void
 nvme_sigbus_fault_sighandler(int signum, siginfo_t *info, void *ctx)
@@ -271,7 +271,7 @@ _nvme_pcie_hotplug_monitor(struct spdk_nvme_probe_ctx *probe_ctx)
 	union spdk_nvme_csts_register csts;
 	struct spdk_nvme_ctrlr_process *proc;
 
-	while (spdk_get_uevent(hotplug_fd, &event) > 0) {
+	while (spdk_get_uevent(g_hotplug_fd, &event) > 0) {
 		if (event.subsystem == SPDK_NVME_UEVENT_SUBSYSTEM_UIO ||
 		    event.subsystem == SPDK_NVME_UEVENT_SUBSYSTEM_VFIO) {
 			if (event.action == SPDK_NVME_UEVENT_ADD) {
@@ -766,13 +766,16 @@ nvme_pcie_ctrlr_scan(struct spdk_nvme_probe_ctx *probe_ctx,
 		enum_ctx.has_pci_addr = true;
 	}
 
-	if (hotplug_fd < 0) {
-		hotplug_fd = spdk_uevent_connect();
-		if (hotplug_fd < 0) {
-			SPDK_DEBUGLOG(SPDK_LOG_NVME, "Failed to open uevent netlink socket\n");
+	/* Only the primary process can monitor hotplug. */
+	if (spdk_process_is_primary()) {
+		if (g_hotplug_fd < 0) {
+			g_hotplug_fd = spdk_uevent_connect();
+			if (g_hotplug_fd < 0) {
+				SPDK_DEBUGLOG(SPDK_LOG_NVME, "Failed to open uevent netlink socket\n");
+			}
+		} else {
+			_nvme_pcie_hotplug_monitor(probe_ctx);
 		}
-	} else {
-		_nvme_pcie_hotplug_monitor(probe_ctx);
 	}
 
 	if (enum_ctx.has_pci_addr == false) {
@@ -1812,6 +1815,79 @@ nvme_pcie_qpair_build_contig_request(struct spdk_nvme_qpair *qpair, struct nvme_
 	return 0;
 }
 
+/**
+ * Build an SGL describing a physically contiguous payload buffer.
+ *
+ * This is more efficient than using PRP because large buffers can be
+ * described this way.
+ */
+static int
+nvme_pcie_qpair_build_contig_hw_sgl_request(struct spdk_nvme_qpair *qpair, struct nvme_request *req,
+		struct nvme_tracker *tr)
+{
+	void *virt_addr;
+	uint64_t phys_addr, mapping_length;
+	uint32_t length;
+	struct spdk_nvme_sgl_descriptor *sgl;
+	uint32_t nseg = 0;
+
+	assert(req->payload_size != 0);
+	assert(nvme_payload_type(&req->payload) == NVME_PAYLOAD_TYPE_CONTIG);
+
+	sgl = tr->u.sgl;
+	req->cmd.psdt = SPDK_NVME_PSDT_SGL_MPTR_CONTIG;
+	req->cmd.dptr.sgl1.unkeyed.subtype = 0;
+
+	length = req->payload_size;
+	virt_addr = req->payload.contig_or_cb_arg + req->payload_offset;
+	mapping_length = length;
+
+	while (length > 0) {
+		if (nseg >= NVME_MAX_SGL_DESCRIPTORS) {
+			nvme_pcie_fail_request_bad_vtophys(qpair, tr);
+			return -EFAULT;
+		}
+
+		phys_addr = spdk_vtophys(virt_addr, &mapping_length);
+		if (phys_addr == SPDK_VTOPHYS_ERROR) {
+			nvme_pcie_fail_request_bad_vtophys(qpair, tr);
+			return -EFAULT;
+		}
+
+		mapping_length = spdk_min(length, mapping_length);
+
+		length -= mapping_length;
+		virt_addr += mapping_length;
+
+		sgl->unkeyed.type = SPDK_NVME_SGL_TYPE_DATA_BLOCK;
+		sgl->unkeyed.length = mapping_length;
+		sgl->address = phys_addr;
+		sgl->unkeyed.subtype = 0;
+
+		sgl++;
+		nseg++;
+	}
+
+	if (nseg == 1) {
+		/*
+		 * The whole transfer can be described by a single SGL descriptor.
+		 *  Use the special case described by the spec where SGL1's type is Data Block.
+		 *  This means the SGL in the tracker is not used at all, so copy the first (and only)
+		 *  SGL element into SGL1.
+		 */
+		req->cmd.dptr.sgl1.unkeyed.type = SPDK_NVME_SGL_TYPE_DATA_BLOCK;
+		req->cmd.dptr.sgl1.address = tr->u.sgl[0].address;
+		req->cmd.dptr.sgl1.unkeyed.length = tr->u.sgl[0].unkeyed.length;
+	} else {
+		/* For now we can only support 1 SGL segment in NVMe controller */
+		req->cmd.dptr.sgl1.unkeyed.type = SPDK_NVME_SGL_TYPE_LAST_SEGMENT;
+		req->cmd.dptr.sgl1.address = tr->prp_sgl_bus_addr;
+		req->cmd.dptr.sgl1.unkeyed.length = nseg * sizeof(struct spdk_nvme_sgl_descriptor);
+	}
+
+	return 0;
+}
+
 /**
  * Build SGL list describing scattered payload buffer.
  */
@@ -1998,7 +2074,14 @@ nvme_pcie_qpair_submit_request(struct spdk_nvme_qpair *qpair, struct nvme_reques
 		/* Null payload - leave PRP fields untouched */
 		rc = 0;
 	} else if (nvme_payload_type(&req->payload) == NVME_PAYLOAD_TYPE_CONTIG) {
-		rc = nvme_pcie_qpair_build_contig_request(qpair, req, tr);
+		/* Some NVME drives can't handle SGL request submitted to the admin qpair
+		 * even if they report SGL support */
+		if ((ctrlr->flags & SPDK_NVME_CTRLR_SGL_SUPPORTED) != 0 &&
+		    !nvme_qpair_is_admin_queue(qpair)) {
+			rc = nvme_pcie_qpair_build_contig_hw_sgl_request(qpair, req, tr);
+		} else {
+			rc = nvme_pcie_qpair_build_contig_request(qpair, req, tr);
+		}
 	} else if (nvme_payload_type(&req->payload) == NVME_PAYLOAD_TYPE_SGL) {
 		if (ctrlr->flags & SPDK_NVME_CTRLR_SGL_SUPPORTED) {
 			rc = nvme_pcie_qpair_build_hw_sgl_request(qpair, req, tr);

lib/nvme/nvme_qpair.c

@@ -458,7 +458,7 @@ spdk_nvme_qpair_process_completions(struct spdk_nvme_qpair *qpair, uint32_t max_
 		 * qpair is not enabled, likely because a controller reset is
 		 *  in progress.
 		 */
-		return 0;
+		return -ENXIO;
 	}
 
 	/* error injection for those queued error requests */

lib/nvme/nvme_rdma.c

@@ -70,7 +70,7 @@
 #define NVME_RDMA_NUM_CM_EVENTS			256
 
 /* CM event processing timeout */
-#define NVME_RDMA_QPAIR_CM_EVENT_TIMEOUT_US	100000
+#define NVME_RDMA_QPAIR_CM_EVENT_TIMEOUT_US	1000000
 
 struct spdk_nvmf_cmd {
 	struct spdk_nvme_cmd cmd;
@@ -1911,7 +1911,16 @@ nvme_rdma_qpair_process_completions(struct spdk_nvme_qpair *qpair,
 	return reaped;
 
 fail:
-	nvme_rdma_qpair_disconnect(qpair);
+	/*
+	 * Since admin queues take the ctrlr_lock before entering this function,
+	 * we can call nvme_rdma_qpair_disconnect. For other qpairs we need
+	 * to call the generic function which will take the lock for us.
+	 */
+	if (nvme_qpair_is_admin_queue(qpair)) {
+		nvme_rdma_qpair_disconnect(qpair);
+	} else {
+		nvme_ctrlr_disconnect_qpair(qpair);
+	}
 	return -ENXIO;
 }
 

lib/nvmf/ctrlr.c

@@ -292,17 +292,9 @@ spdk_nvmf_ctrlr_create(struct spdk_nvmf_subsystem *subsystem,
 	ctrlr->feat.volatile_write_cache.bits.wce = 1;
 
 	if (ctrlr->subsys->subtype == SPDK_NVMF_SUBTYPE_DISCOVERY) {
-		/* Don't accept keep-alive timeout for discovery controllers */
-		if (ctrlr->feat.keep_alive_timer.bits.kato != 0) {
-			SPDK_ERRLOG("Discovery controller don't accept keep-alive timeout\n");
-			spdk_bit_array_free(&ctrlr->qpair_mask);
-			free(ctrlr);
-			return NULL;
-		}
-
 		/*
-		 * Discovery controllers use some arbitrary high value in order
-		 * to cleanup stale discovery sessions
+		 * If keep-alive timeout is not set, discovery controllers use some
+		 * arbitrary high value in order to cleanup stale discovery sessions
 		 *
 		 * From the 1.0a nvme-of spec:
 		 * "The Keep Alive command is reserved for
@@ -314,7 +306,9 @@ spdk_nvmf_ctrlr_create(struct spdk_nvmf_subsystem *subsystem,
 		 * actions for Keep Alive Timer expiration".
 		 * kato is in millisecond.
 		 */
-		ctrlr->feat.keep_alive_timer.bits.kato = NVMF_DISC_KATO_IN_MS;
+		if (ctrlr->feat.keep_alive_timer.bits.kato == 0) {
+			ctrlr->feat.keep_alive_timer.bits.kato = NVMF_DISC_KATO_IN_MS;
+		}
 	}
 
 	/* Subtract 1 for admin queue, 1 for 0's based */

lib/nvmf/rdma.c

@@ -333,6 +333,15 @@ struct spdk_nvmf_rdma_resources {
 	STAILQ_HEAD(, spdk_nvmf_rdma_request)	free_queue;
 };
 
+typedef void (*spdk_nvmf_rdma_qpair_ibv_event)(struct spdk_nvmf_rdma_qpair *rqpair);
+
+struct spdk_nvmf_rdma_ibv_event_ctx {
+	struct spdk_nvmf_rdma_qpair			*rqpair;
+	spdk_nvmf_rdma_qpair_ibv_event			cb_fn;
+	/* Link to other ibv events associated with this qpair */
+	STAILQ_ENTRY(spdk_nvmf_rdma_ibv_event_ctx)	link;
+};
+
 struct spdk_nvmf_rdma_qpair {
 	struct spdk_nvmf_qpair			qpair;
 
@@ -402,6 +411,9 @@ struct spdk_nvmf_rdma_qpair {
 
 	struct spdk_poller			*destruct_poller;
 
+	/* List of ibv async events */
+	STAILQ_HEAD(, spdk_nvmf_rdma_ibv_event_ctx)	ibv_events;
+
 	/* There are several ways a disconnect can start on a qpair
 	 * and they are not all mutually exclusive. It is important
 	 * that we only initialize one of these paths.
@@ -901,6 +913,17 @@ cleanup:
 	return NULL;
 }
 
+static void
+spdk_nvmf_rdma_qpair_clean_ibv_events(struct spdk_nvmf_rdma_qpair *rqpair)
+{
+	struct spdk_nvmf_rdma_ibv_event_ctx *ctx, *tctx;
+	STAILQ_FOREACH_SAFE(ctx, &rqpair->ibv_events, link, tctx) {
+		ctx->rqpair = NULL;
+		/* Memory allocated for ctx is freed in spdk_nvmf_rdma_qpair_process_ibv_event */
+		STAILQ_REMOVE(&rqpair->ibv_events, ctx, spdk_nvmf_rdma_ibv_event_ctx, link);
+	}
+}
+
 static void
 spdk_nvmf_rdma_qpair_destroy(struct spdk_nvmf_rdma_qpair *rqpair)
 {
@@ -926,7 +949,7 @@ spdk_nvmf_rdma_qpair_destroy(struct spdk_nvmf_rdma_qpair *rqpair)
 			/* Drop all received but unprocessed commands for this queue and return them to SRQ */
 			STAILQ_FOREACH_SAFE(rdma_recv, &rqpair->resources->incoming_queue, link, recv_tmp) {
 				if (rqpair == rdma_recv->qpair) {
-					STAILQ_REMOVE_HEAD(&rqpair->resources->incoming_queue, link);
+					STAILQ_REMOVE(&rqpair->resources->incoming_queue, rdma_recv, spdk_nvmf_rdma_recv, link);
 					rc = ibv_post_srq_recv(rqpair->srq, &rdma_recv->wr, &bad_recv_wr);
 					if (rc) {
 						SPDK_ERRLOG("Unable to re-post rx descriptor\n");
@@ -951,6 +974,8 @@ spdk_nvmf_rdma_qpair_destroy(struct spdk_nvmf_rdma_qpair *rqpair)
 		nvmf_rdma_resources_destroy(rqpair->resources);
 	}
 
+	spdk_nvmf_rdma_qpair_clean_ibv_events(rqpair);
+
 	free(rqpair);
 }
 
@@ -1334,6 +1359,7 @@ nvmf_rdma_connect(struct spdk_nvmf_transport *transport, struct rdma_cm_event *e
 	rqpair->cm_id = event->id;
 	rqpair->listen_id = event->listen_id;
 	rqpair->qpair.transport = transport;
+	STAILQ_INIT(&rqpair->ibv_events);
 	/* use qid from the private data to determine the qpair type
 	   qid will be set to the appropriate value when the controller is created */
 	rqpair->qpair.qid = private_data->qid;
@@ -2963,8 +2989,6 @@ nvmf_rdma_disconnect(struct rdma_cm_event *evt)
 
 	spdk_trace_record(TRACE_RDMA_QP_DISCONNECT, 0, 0, (uintptr_t)rqpair->cm_id, 0);
 
-	spdk_nvmf_rdma_update_ibv_state(rqpair);
-
 	spdk_nvmf_rdma_start_disconnect(rqpair);
 
 	return 0;
@@ -2991,15 +3015,6 @@ static const char *CM_EVENT_STR[] = {
 };
 #endif /* DEBUG */
 
-static void
-nvmf_rdma_handle_last_wqe_reached(void *ctx)
-{
-	struct spdk_nvmf_rdma_qpair *rqpair = ctx;
-	rqpair->last_wqe_reached = true;
-
-	nvmf_rdma_destroy_drained_qpair(rqpair);
-}
-
 static void
 spdk_nvmf_process_cm_event(struct spdk_nvmf_transport *transport, new_qpair_fn cb_fn, void *cb_arg)
 {
@@ -3080,13 +3095,69 @@ spdk_nvmf_process_cm_event(struct spdk_nvmf_transport *transport, new_qpair_fn c
 	}
 }
 
+static void
+nvmf_rdma_handle_qp_fatal(struct spdk_nvmf_rdma_qpair *rqpair)
+{
+	spdk_nvmf_rdma_update_ibv_state(rqpair);
+	spdk_nvmf_rdma_start_disconnect(rqpair);
+}
+
+static void
+nvmf_rdma_handle_last_wqe_reached(struct spdk_nvmf_rdma_qpair *rqpair)
+{
+	rqpair->last_wqe_reached = true;
+	nvmf_rdma_destroy_drained_qpair(rqpair);
+}
+
+static void
+nvmf_rdma_handle_sq_drained(struct spdk_nvmf_rdma_qpair *rqpair)
+{
+	spdk_nvmf_rdma_start_disconnect(rqpair);
+}
+
+static void
+spdk_nvmf_rdma_qpair_process_ibv_event(void *ctx)
+{
+	struct spdk_nvmf_rdma_ibv_event_ctx *event_ctx = ctx;
+
+	if (event_ctx->rqpair) {
+		STAILQ_REMOVE(&event_ctx->rqpair->ibv_events, event_ctx, spdk_nvmf_rdma_ibv_event_ctx, link);
+		if (event_ctx->cb_fn) {
+			event_ctx->cb_fn(event_ctx->rqpair);
+		}
+	}
+	free(event_ctx);
+}
+
+static int
+spdk_nvmf_rdma_send_qpair_async_event(struct spdk_nvmf_rdma_qpair *rqpair,
+				      spdk_nvmf_rdma_qpair_ibv_event fn)
+{
+	struct spdk_nvmf_rdma_ibv_event_ctx *ctx;
+
+	if (!rqpair->qpair.group) {
+		return EINVAL;
+	}
+
+	ctx = calloc(1, sizeof(*ctx));
+	if (!ctx) {
+		return ENOMEM;
+	}
+
+	ctx->rqpair = rqpair;
+	ctx->cb_fn = fn;
+	STAILQ_INSERT_TAIL(&rqpair->ibv_events, ctx, link);
+
+	return spdk_thread_send_msg(rqpair->qpair.group->thread, spdk_nvmf_rdma_qpair_process_ibv_event,
+				    ctx);
+}
+
 static void
 spdk_nvmf_process_ib_event(struct spdk_nvmf_rdma_device *device)
 {
 	int				rc;
 	struct spdk_nvmf_rdma_qpair	*rqpair = NULL;
 	struct ibv_async_event		event;
-	enum ibv_qp_state		state;
 
 	rc = ibv_get_async_event(device->context, &event);
 
@@ -3102,35 +3173,32 @@ spdk_nvmf_process_ib_event(struct spdk_nvmf_rdma_device *device)
 		SPDK_ERRLOG("Fatal event received for rqpair %p\n", rqpair);
 		spdk_trace_record(TRACE_RDMA_IBV_ASYNC_EVENT, 0, 0,
 				  (uintptr_t)rqpair->cm_id, event.event_type);
-		spdk_nvmf_rdma_update_ibv_state(rqpair);
-		spdk_nvmf_rdma_start_disconnect(rqpair);
+		if (spdk_nvmf_rdma_send_qpair_async_event(rqpair, nvmf_rdma_handle_qp_fatal)) {
+			SPDK_ERRLOG("Failed to send QP_FATAL event for rqpair %p\n", rqpair);
+			nvmf_rdma_handle_qp_fatal(rqpair);
+		}
 		break;
 	case IBV_EVENT_QP_LAST_WQE_REACHED:
 		/* This event only occurs for shared receive queues. */
 		rqpair = event.element.qp->qp_context;
 		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Last WQE reached event received for rqpair %p\n", rqpair);
-		/* This must be handled on the polling thread if it exists. Otherwise the timeout will catch it. */
-		if (rqpair->qpair.group) {
-			spdk_thread_send_msg(rqpair->qpair.group->thread, nvmf_rdma_handle_last_wqe_reached, rqpair);
-		} else {
-			SPDK_ERRLOG("Unable to destroy the qpair %p since it does not have a poll group.\n", rqpair);
+		if (spdk_nvmf_rdma_send_qpair_async_event(rqpair, nvmf_rdma_handle_last_wqe_reached)) {
+			SPDK_ERRLOG("Failed to send LAST_WQE_REACHED event for rqpair %p\n", rqpair);
 			rqpair->last_wqe_reached = true;
 		}
-
 		break;
 	case IBV_EVENT_SQ_DRAINED:
 		/* This event occurs frequently in both error and non-error states.
-		 * Check if the qpair is in an error state before sending a message.
-		 * Note that we're not on the correct thread to access the qpair, but
-		 * the operations that the below calls make all happen to be thread
-		 * safe. */
+		 * Check if the qpair is in an error state before sending a message. */
 		rqpair = event.element.qp->qp_context;
 		SPDK_DEBUGLOG(SPDK_LOG_RDMA, "Last sq drained event received for rqpair %p\n", rqpair);
 		spdk_trace_record(TRACE_RDMA_IBV_ASYNC_EVENT, 0, 0,
 				  (uintptr_t)rqpair->cm_id, event.event_type);
-		state = spdk_nvmf_rdma_update_ibv_state(rqpair);
-		if (state == IBV_QPS_ERR) {
-			spdk_nvmf_rdma_start_disconnect(rqpair);
+		if (spdk_nvmf_rdma_update_ibv_state(rqpair) == IBV_QPS_ERR) {
+			if (spdk_nvmf_rdma_send_qpair_async_event(rqpair, nvmf_rdma_handle_sq_drained)) {
+				SPDK_ERRLOG("Failed to send SQ_DRAINED event for rqpair %p\n", rqpair);
+				nvmf_rdma_handle_sq_drained(rqpair);
+			}
 		}
 		break;
 	case IBV_EVENT_QP_REQ_ERR:

lib/thread/thread.c

@@ -648,17 +648,14 @@ spdk_thread_get_stats(struct spdk_thread_stats *stats)
 	return 0;
 }
 
-void
+int
 spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx)
 {
 	struct spdk_thread *local_thread;
 	struct spdk_msg *msg;
 	int rc;
 
-	if (!thread) {
-		assert(false);
-		return;
-	}
+	assert(thread != NULL);
 
 	local_thread = _get_thread();
 
@@ -675,8 +672,8 @@ spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx
 	if (msg == NULL) {
 		msg = spdk_mempool_get(g_spdk_msg_mempool);
 		if (!msg) {
-			assert(false);
-			return;
+			SPDK_ERRLOG("msg could not be allocated\n");
+			return -ENOMEM;
 		}
 	}
 
@@ -685,10 +682,12 @@ spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx
 
 	rc = spdk_ring_enqueue(thread->messages, (void **)&msg, 1, NULL);
 	if (rc != 1) {
-		assert(false);
+		SPDK_ERRLOG("msg could not be enqueued\n");
 		spdk_mempool_put(g_spdk_msg_mempool, msg);
-		return;
+		return -EIO;
 	}
+
+	return 0;
 }
 
 struct spdk_poller *
@@ -1200,6 +1199,7 @@ spdk_for_each_channel(void *io_device, spdk_channel_msg fn, void *ctx,
 	struct spdk_thread *thread;
 	struct spdk_io_channel *ch;
 	struct spdk_io_channel_iter *i;
+	int rc __attribute__((unused));
 
 	i = calloc(1, sizeof(*i));
 	if (!i) {
@@ -1223,7 +1223,8 @@ spdk_for_each_channel(void *io_device, spdk_channel_msg fn, void *ctx,
 				i->cur_thread = thread;
 				i->ch = ch;
 				pthread_mutex_unlock(&g_devlist_mutex);
-				spdk_thread_send_msg(thread, _call_channel, i);
+				rc = spdk_thread_send_msg(thread, _call_channel, i);
+				assert(rc == 0);
 				return;
 			}
 		}
@@ -1231,7 +1232,8 @@ spdk_for_each_channel(void *io_device, spdk_channel_msg fn, void *ctx,
 
 	pthread_mutex_unlock(&g_devlist_mutex);
 
-	spdk_thread_send_msg(i->orig_thread, _call_completion, i);
+	rc = spdk_thread_send_msg(i->orig_thread, _call_completion, i);
+	assert(rc == 0);
 }
 
 void

lib/vmd/vmd.c

@@ -869,7 +869,7 @@ vmd_scan_single_bus(struct vmd_pci_bus *bus, struct vmd_pci_device *parent_bridg
 			new_bus->self = new_dev;
 			new_dev->bus_object = new_bus;
 
-			if (slot_cap.bit_field.hotplug_capable &&
+			if (slot_cap.bit_field.hotplug_capable && new_dev->pcie_cap != NULL &&
 			    new_dev->pcie_cap->express_cap_register.bit_field.slot_implemented) {
 				new_bus->hotplug_buses = vmd_get_hotplug_bus_numbers(new_dev);
 				new_bus->subordinate_bus += new_bus->hotplug_buses;

module/bdev/nvme/bdev_nvme.c

@@ -210,13 +210,6 @@ bdev_nvme_poll_adminq(void *arg)
 {
 	struct spdk_nvme_ctrlr *ctrlr = arg;
 
-	/* Process io messages that were passed from non-polled mode threads
-	 * to this ctrlr. This is used as part of nvme cuse support for surfacing
-	 * /dev nodes that can be used by standard Linux management applications
-	 * like nvme-cli.
-	 */
-	spdk_nvme_io_msg_process(ctrlr);
-
 	return spdk_nvme_ctrlr_process_admin_completions(ctrlr);
 }
 

module/bdev/nvme/bdev_nvme_cuse_rpc.c

@@ -44,19 +44,16 @@
 
 struct rpc_nvme_cuse_register {
 	char *name;
-	char *dev_path;
 };
 
 static void
 free_rpc_nvme_cuse_register(struct rpc_nvme_cuse_register *req)
 {
 	free(req->name);
-	free(req->dev_path);
 }
 
 static const struct spdk_json_object_decoder rpc_nvme_cuse_register_decoders[] = {
 	{"name", offsetof(struct rpc_nvme_cuse_register, name), spdk_json_decode_string},
-	{"dev_path", offsetof(struct rpc_nvme_cuse_register, dev_path), spdk_json_decode_string},
 };
 
 static void
@@ -84,10 +81,11 @@ spdk_rpc_nvme_cuse_register(struct spdk_jsonrpc_request *request,
 		goto cleanup;
 	}
 
-	rc = spdk_nvme_cuse_register(bdev_ctrlr->ctrlr, req.dev_path);
+	rc = spdk_nvme_cuse_register(bdev_ctrlr->ctrlr);
 	if (rc) {
 		SPDK_ERRLOG("Failed to register CUSE devices\n");
 		spdk_jsonrpc_send_error_response(request, -rc, spdk_strerror(rc));
+		goto cleanup;
 	}
 
 	w = spdk_jsonrpc_begin_result(request);

pkg/spdk.spec

@@ -2,12 +2,12 @@
 %bcond_with doc
 
 Name: spdk
-Version: master
+Version: 19.10.1
 Release: 0%{?dist}
 Epoch: 0
 URL: http://spdk.io
 
-Source: https://github.com/spdk/spdk/archive/master.tar.gz
+Source: https://github.com/spdk/spdk/archive/v19.10.1.tar.gz
 Summary: Set of libraries and utilities for high performance user-mode storage
 
 %define package_version %{epoch}:%{version}-%{release}

scripts/rpc.py

@@ -450,15 +450,12 @@ if __name__ == "__main__":
 
     def bdev_nvme_cuse_register(args):
         rpc.bdev.bdev_nvme_cuse_register(args.client,
-                                         name=args.name,
-                                         dev_path=args.dev_path)
+                                         name=args.name)
 
     p = subparsers.add_parser('bdev_nvme_cuse_register',
                               help='Register CUSE devices on NVMe controller')
     p.add_argument('-n', '--name',
                    help='Name of the NVMe controller. Example: Nvme0', required=True)
-    p.add_argument('-p', '--dev_path',
-                   help='CUSE dev path including prefix: e.g. spdk/nvme0 will result: /dev/spdk/nvme0n1', required=True)
     p.set_defaults(func=bdev_nvme_cuse_register)
 
     def bdev_nvme_cuse_unregister(args):

scripts/rpc/bdev.py

@@ -480,15 +480,13 @@ def bdev_nvme_detach_controller(client, name):
     return client.call('bdev_nvme_detach_controller', params)
 
 
-def bdev_nvme_cuse_register(client, name, dev_path):
+def bdev_nvme_cuse_register(client, name):
     """Register CUSE devices on NVMe controller.
 
     Args:
         name: Name of the operating NVMe controller
-        dev_path: CUSE dev path with dev prefix
     """
-    params = {'name': name,
-              'dev_path': dev_path}
+    params = {'name': name}
 
     return client.call('bdev_nvme_cuse_register', params)
 

test/common/autotest_common.sh

@@ -94,6 +94,8 @@ echo "leak:spdk_fs_alloc_thread_ctx" >> "$asan_suppression_file"
 echo "leak:/usr/src/fio/parse.c" >> "$asan_suppression_file"
 echo "leak:/usr/src/fio/iolog.c" >> "$asan_suppression_file"
 echo "leak:/usr/src/fio/init.c" >> "$asan_suppression_file"
+echo "leak:fio_memalign" >> "$asan_suppression_file"
+echo "leak:spdk_fio_io_u_init" >> "$asan_suppression_file"
 
 # Suppress leaks in libiscsi
 echo "leak:libiscsi.so" >> "$asan_suppression_file"

test/nvme/spdk_nvme_cli_cuse.sh

@@ -19,10 +19,14 @@ waitforlisten $spdk_tgt_pid
 bdf=$(iter_pci_class_code 01 08 02 | head -1)
 
 $rpc_py bdev_nvme_attach_controller -b Nvme0 -t PCIe -a ${bdf}
-$rpc_py bdev_nvme_cuse_register -n Nvme0 -p spdk/nvme0
+$rpc_py bdev_nvme_cuse_register -n Nvme0
 
 sleep 5
 
+if [ ! -c /dev/spdk/nvme0 ]; then
+	return 1
+fi
+
 $rpc_py bdev_get_bdevs
 $rpc_py bdev_nvme_get_controllers
 
@@ -43,6 +47,31 @@ for ctrlr in $(ls /dev/spdk/nvme?); do
 	${NVME_CMD} reset $ctrlr
 done
 
+if [ ! -c /dev/spdk/nvme0 ]; then
+	return 1
+fi
+
+$rpc_py bdev_nvme_cuse_unregister -n Nvme0
+sleep 1
+if [ -c /dev/spdk/nvme0 ]; then
+	return 1
+fi
+
+$rpc_py bdev_nvme_cuse_register -n Nvme0
+sleep 1
+
+if [ ! -c /dev/spdk/nvme0 ]; then
+	return 1
+fi
+
+$rpc_py bdev_nvme_cuse_unregister -n Nvme0
+sleep 1
+if [ -c /dev/spdk/nvme0 ]; then
+	return 1
+fi
+
+$rpc_py bdev_nvme_detach_controller Nvme0
+
 trap - SIGINT SIGTERM EXIT
 kill $spdk_tgt_pid
 

test/unit/lib/bdev/bdev_raid.c/bdev_raid_ut.c

@@ -407,12 +407,6 @@ spdk_for_each_thread(spdk_msg_fn fn, void *ctx, spdk_msg_fn cpl)
 	cpl(ctx);
 }
 
-void
-spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx)
-{
-	fn(ctx);
-}
-
 void
 spdk_bdev_free_io(struct spdk_bdev_io *bdev_io)
 {

test/unit/lib/bdev/compress.c/compress_ut.c

@@ -64,7 +64,6 @@ static struct rte_mbuf g_expected_src_mbufs[UT_MBUFS_PER_OP_BOUND_TEST];
 static struct rte_mbuf g_expected_dst_mbufs[UT_MBUFS_PER_OP];
 struct comp_bdev_io *g_io_ctx;
 struct comp_io_channel *g_comp_ch;
-struct rte_config *g_test_config;
 
 /* Those functions are defined as static inline in DPDK, so we can't
  * mock them straight away. We use defines to redirect them into
@@ -296,7 +295,6 @@ DEFINE_STUB(spdk_reduce_vol_get_params, const struct spdk_reduce_vol_params *,
 
 /* DPDK stubs */
 DEFINE_STUB(rte_socket_id, unsigned, (void), 0);
-DEFINE_STUB(rte_eal_get_configuration, struct rte_config *, (void), NULL);
 DEFINE_STUB(rte_vdev_init, int, (const char *name, const char *args), 0);
 DEFINE_STUB_V(rte_comp_op_free, (struct rte_comp_op *op));
 DEFINE_STUB(rte_comp_op_alloc, struct rte_comp_op *, (struct rte_mempool *mempool), NULL);
@@ -569,9 +567,6 @@ test_setup(void)
 	g_io_ctx->comp_bdev = &g_comp_bdev;
 	g_comp_bdev.device_qp = &g_device_qp;
 
-	g_test_config = calloc(1, sizeof(struct rte_config));
-	g_test_config->lcore_count = 1;
-
 	for (i = 0; i < UT_MBUFS_PER_OP_BOUND_TEST - 1; i++) {
 		g_expected_src_mbufs[i].next = &g_expected_src_mbufs[i + 1];
 	}
@@ -601,7 +596,6 @@ test_cleanup(void)
 	free(g_bdev_io->u.bdev.iovs);
 	free(g_bdev_io);
 	free(g_io_ch);
-	free(g_test_config);
 	return 0;
 }
 
@@ -1002,7 +996,6 @@ test_initdrivers(void)
 	int rc;
 
 	/* test return values from rte_vdev_init() */
-	MOCK_SET(rte_eal_get_configuration, g_test_config);
 	MOCK_SET(rte_vdev_init, -EEXIST);
 	rc = vbdev_init_compress_drivers();
 	/* This is not an error condition, we already have one */

test/unit/lib/bdev/crypto.c/crypto_ut.c

@@ -136,6 +136,13 @@ mock_rte_crypto_op_attach_sym_session(struct rte_crypto_op *op,
 	return ut_rte_crypto_op_attach_sym_session;
 }
 
+#define rte_lcore_count mock_rte_lcore_count
+static inline unsigned
+mock_rte_lcore_count(void)
+{
+	return 1;
+}
+
 #include "bdev/crypto/vbdev_crypto.c"
 
 /* SPDK stubs */
@@ -168,7 +175,6 @@ DEFINE_STUB(spdk_bdev_register, int, (struct spdk_bdev *vbdev), 0);
 
 /* DPDK stubs */
 DEFINE_STUB(rte_cryptodev_count, uint8_t, (void), 0);
-DEFINE_STUB(rte_eal_get_configuration, struct rte_config *, (void), NULL);
 DEFINE_STUB_V(rte_mempool_free, (struct rte_mempool *mp));
 DEFINE_STUB(rte_mempool_create, struct rte_mempool *, (const char *name, unsigned n,
 		unsigned elt_size,
@@ -214,7 +220,6 @@ struct crypto_io_channel *g_crypto_ch;
 struct spdk_io_channel *g_io_ch;
 struct vbdev_dev g_device;
 struct vbdev_crypto g_crypto_bdev;
-struct rte_config *g_test_config;
 struct device_qp g_dev_qp;
 
 void
@@ -319,8 +324,6 @@ test_setup(void)
 	g_io_ctx->crypto_ch = g_crypto_ch;
 	g_io_ctx->crypto_bdev = &g_crypto_bdev;
 	g_crypto_ch->device_qp = &g_dev_qp;
-	g_test_config = calloc(1, sizeof(struct rte_config));
-	g_test_config->lcore_count = 1;
 	TAILQ_INIT(&g_crypto_ch->pending_cry_ios);
 
 	/* Allocate a real mbuf pool so we can test error paths */
@@ -350,7 +353,6 @@ test_cleanup(void)
 {
 	int i;
 
-	free(g_test_config);
 	spdk_mempool_free(g_mbuf_mp);
 	for (i = 0; i < MAX_TEST_BLOCKS; i++) {
 		free(g_test_crypto_ops[i]);
@@ -719,7 +721,6 @@ test_initdrivers(void)
 	g_mbuf_mp = NULL;
 
 	/* No drivers available, not an error though */
-	MOCK_SET(rte_eal_get_configuration, g_test_config);
 	MOCK_SET(rte_cryptodev_count, 0);
 	rc = vbdev_crypto_init_crypto_drivers();
 	CU_ASSERT(rc == 0);

test/unit/lib/blob/blob.c/blob_ut.c

@@ -4688,6 +4688,139 @@ blob_thin_prov_rw(void)
 	g_blobid = 0;
 }
 
+static void
+blob_thin_prov_rle(void)
+{
+	static const uint8_t zero[10 * 4096] = { 0 };
+	struct spdk_blob_store *bs;
+	struct spdk_bs_dev *dev;
+	struct spdk_blob *blob;
+	struct spdk_io_channel *channel;
+	struct spdk_blob_opts opts;
+	spdk_blob_id blobid;
+	uint64_t free_clusters;
+	uint64_t page_size;
+	uint8_t payload_read[10 * 4096];
+	uint8_t payload_write[10 * 4096];
+	uint64_t write_bytes;
+	uint64_t read_bytes;
+	uint64_t io_unit;
+
+	dev = init_dev();
+
+	spdk_bs_init(dev, NULL, bs_op_with_handle_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	SPDK_CU_ASSERT_FATAL(g_bs != NULL);
+	bs = g_bs;
+	free_clusters = spdk_bs_free_cluster_count(bs);
+	page_size = spdk_bs_get_page_size(bs);
+
+	spdk_blob_opts_init(&opts);
+	opts.thin_provision = true;
+	opts.num_clusters = 5;
+
+	spdk_bs_create_blob_ext(bs, &opts, blob_op_with_id_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	CU_ASSERT(g_blobid != SPDK_BLOBID_INVALID);
+	CU_ASSERT(free_clusters == spdk_bs_free_cluster_count(bs));
+	blobid = g_blobid;
+
+	spdk_bs_open_blob(bs, blobid, blob_op_with_handle_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	SPDK_CU_ASSERT_FATAL(g_blob != NULL);
+	blob = g_blob;
+
+	channel = spdk_bs_alloc_io_channel(bs);
+	CU_ASSERT(channel != NULL);
+
+	/* Target specifically second cluster in a blob as first allocation */
+	io_unit = _spdk_bs_cluster_to_page(bs, 1) * _spdk_bs_io_unit_per_page(bs);
+
+	/* Payload should be all zeros from unallocated clusters */
+	memset(payload_read, 0xFF, sizeof(payload_read));
+	spdk_blob_io_read(blob, channel, payload_read, io_unit, 10, blob_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	CU_ASSERT(memcmp(zero, payload_read, 10 * 4096) == 0);
+
+	write_bytes = g_dev_write_bytes;
+	read_bytes = g_dev_read_bytes;
+
+	/* Issue write to second cluster in a blob */
+	memset(payload_write, 0xE5, sizeof(payload_write));
+	spdk_blob_io_write(blob, channel, payload_write, io_unit, 10, blob_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	CU_ASSERT(free_clusters - 1 == spdk_bs_free_cluster_count(bs));
+	/* For thin-provisioned blob we need to write 10 pages plus one page metadata and
+	 * read 0 bytes */
+	CU_ASSERT(g_dev_write_bytes - write_bytes == page_size * 11);
+	CU_ASSERT(g_dev_read_bytes - read_bytes == 0);
+
+	spdk_blob_io_read(blob, channel, payload_read, io_unit, 10, blob_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	CU_ASSERT(memcmp(payload_write, payload_read, 10 * 4096) == 0);
+
+	spdk_bs_free_io_channel(channel);
+	poll_threads();
+
+	spdk_blob_close(blob, blob_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+
+	/* Unload the blob store */
+	spdk_bs_unload(g_bs, bs_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	g_bs = NULL;
+	g_blob = NULL;
+	g_blobid = 0;
+
+	/* Load an existing blob store */
+	dev = init_dev();
+	spdk_bs_load(dev, NULL, bs_op_with_handle_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	SPDK_CU_ASSERT_FATAL(g_bs != NULL);
+
+	bs = g_bs;
+
+	spdk_bs_open_blob(g_bs, blobid, blob_op_with_handle_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	SPDK_CU_ASSERT_FATAL(g_blob != NULL);
+	blob = g_blob;
+
+	channel = spdk_bs_alloc_io_channel(bs);
+	CU_ASSERT(channel != NULL);
+
+	/* Read second cluster after blob reload to confirm data written */
+	spdk_blob_io_read(blob, channel, payload_read, io_unit, 10, blob_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	CU_ASSERT(memcmp(payload_write, payload_read, 10 * 4096) == 0);
+
+	spdk_bs_free_io_channel(channel);
+	poll_threads();
+
+	spdk_blob_close(blob, blob_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+
+	spdk_bs_delete_blob(bs, blobid, blob_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+
+	spdk_bs_unload(g_bs, bs_op_complete, NULL);
+	poll_threads();
+	CU_ASSERT(g_bserrno == 0);
+	g_bs = NULL;
+}
+
 static void
 blob_thin_prov_rw_iov(void)
 {
@@ -7447,6 +7580,7 @@ int main(int argc, char **argv)
 		CU_add_test(suite, "blob_thin_prov_alloc", blob_thin_prov_alloc) == NULL ||
 		CU_add_test(suite, "blob_insert_cluster_msg", blob_insert_cluster_msg) == NULL ||
 		CU_add_test(suite, "blob_thin_prov_rw", blob_thin_prov_rw) == NULL ||
+		CU_add_test(suite, "blob_thin_prov_rle", blob_thin_prov_rle) == NULL ||
 		CU_add_test(suite, "blob_thin_prov_rw_iov", blob_thin_prov_rw_iov) == NULL ||
 		CU_add_test(suite, "bs_load_iter", bs_load_iter) == NULL ||
 		CU_add_test(suite, "blob_snapshot_rw", blob_snapshot_rw) == NULL ||

test/unit/lib/blobfs/blobfs_bdev.c/blobfs_bdev_ut.c

@@ -145,10 +145,11 @@ spdk_bdev_close(struct spdk_bdev_desc *desc)
 {
 }
 
-void
+int
 spdk_thread_send_msg(const struct spdk_thread *thread, spdk_msg_fn fn, void *ctx)
 {
 	fn(ctx);
+	return 0;
 }
 
 struct spdk_thread *

test/unit/lib/nvme/nvme.c/nvme_ut.c

@@ -60,8 +60,7 @@ DEFINE_STUB(nvme_transport_ctrlr_construct, struct spdk_nvme_ctrlr *,
 	    (const struct spdk_nvme_transport_id *trid,
 	     const struct spdk_nvme_ctrlr_opts *opts,
 	     void *devhandle), NULL);
-DEFINE_STUB_V(nvme_io_msg_ctrlr_stop, (struct spdk_nvme_ctrlr *ctrlr,
-				       struct nvme_io_msg_producer *io_msg_producer, bool shutdown));
+DEFINE_STUB_V(nvme_io_msg_ctrlr_detach, (struct spdk_nvme_ctrlr *ctrlr));
 
 static bool ut_destruct_called = false;
 void

test/unit/lib/nvme/nvme_pcie.c/nvme_pcie_ut.c

@@ -35,6 +35,7 @@
 
 #include "spdk_cunit.h"
 
+#define UNIT_TEST_NO_VTOPHYS
 #include "common/lib/test_env.c"
 
 #include "nvme/nvme_pcie.c"
@@ -232,24 +233,6 @@ nvme_get_quirks(const struct spdk_pci_id *id)
 	abort();
 }
 
-bool
-nvme_completion_is_retry(const struct spdk_nvme_cpl *cpl)
-{
-	abort();
-}
-
-void
-spdk_nvme_qpair_print_command(struct spdk_nvme_qpair *qpair, struct spdk_nvme_cmd *cmd)
-{
-	abort();
-}
-
-void
-spdk_nvme_qpair_print_completion(struct spdk_nvme_qpair *qpair, struct spdk_nvme_cpl *cpl)
-{
-	abort();
-}
-
 int
 nvme_qpair_submit_request(struct spdk_nvme_qpair *qpair, struct nvme_request *req)
 {
@@ -570,98 +553,31 @@ test_hw_sgl_req(void)
 	nvme_free_request(req);
 }
 
-static void test_nvme_qpair_abort_reqs(void)
-{
-	struct spdk_nvme_qpair		qpair = {};
-	struct nvme_request		*req = NULL;
-	struct spdk_nvme_ctrlr		ctrlr = {};
-	struct nvme_tracker		*tr_temp;
-
-	prepare_submit_request_test(&qpair, &ctrlr);
-
-	tr_temp = TAILQ_FIRST(&qpair.free_tr);
-	SPDK_CU_ASSERT_FATAL(tr_temp != NULL);
-	TAILQ_REMOVE(&qpair.free_tr, tr_temp, tq_list);
-	tr_temp->req = nvme_allocate_request_null(expected_failure_callback, NULL);
-	SPDK_CU_ASSERT_FATAL(tr_temp->req != NULL);
-	tr_temp->req->cmd.cid = tr_temp->cid;
-
-	TAILQ_INSERT_HEAD(&qpair.outstanding_tr, tr_temp, tq_list);
-	nvme_qpair_abort_reqs(&qpair, true);
-	CU_ASSERT_TRUE(TAILQ_EMPTY(&qpair.outstanding_tr));
-
-	req = nvme_allocate_request_null(expected_failure_callback, NULL);
-	SPDK_CU_ASSERT_FATAL(req != NULL);
-
-	STAILQ_INSERT_HEAD(&qpair.queued_req, req, stailq);
-	nvme_qpair_abort_reqs(&qpair, true);
-	CU_ASSERT_TRUE(STAILQ_EMPTY(&qpair.queued_req));
-
-	cleanup_submit_request_test(&qpair);
-}
-
-static void
-test_nvme_qpair_process_completions_limit(void)
-{
-	struct spdk_nvme_qpair		qpair = {};
-	struct spdk_nvme_ctrlr		ctrlr = {};
-
-	prepare_submit_request_test(&qpair, &ctrlr);
-	qpair.is_enabled = true;
-
-	/* Insert 4 entries into the completion queue */
-	CU_ASSERT(qpair.cq_head == 0);
-	ut_insert_cq_entry(&qpair, 0);
-	ut_insert_cq_entry(&qpair, 1);
-	ut_insert_cq_entry(&qpair, 2);
-	ut_insert_cq_entry(&qpair, 3);
-
-	/* This should only process 2 completions, and 2 should be left in the queue */
-	spdk_nvme_qpair_process_completions(&qpair, 2);
-	CU_ASSERT(qpair.cq_head == 2);
-
-	/* This should only process 1 completion, and 1 should be left in the queue */
-	spdk_nvme_qpair_process_completions(&qpair, 1);
-	CU_ASSERT(qpair.cq_head == 3);
-
-	/* This should process the remaining completion */
-	spdk_nvme_qpair_process_completions(&qpair, 5);
-	CU_ASSERT(qpair.cq_head == 4);
-
-	cleanup_submit_request_test(&qpair);
-}
-
-static void test_nvme_qpair_destroy(void)
-{
-	struct spdk_nvme_qpair		qpair = {};
-	struct spdk_nvme_ctrlr		ctrlr = {};
-	struct nvme_tracker		*tr_temp;
-
-	memset(&ctrlr, 0, sizeof(ctrlr));
-	TAILQ_INIT(&ctrlr.free_io_qpairs);
-	TAILQ_INIT(&ctrlr.active_io_qpairs);
-	TAILQ_INIT(&ctrlr.active_procs);
-
-	nvme_qpair_init(&qpair, 1, 128, &ctrlr);
-	nvme_qpair_destroy(&qpair);
-
-
-	nvme_qpair_init(&qpair, 0, 128, &ctrlr);
-	tr_temp = TAILQ_FIRST(&qpair.free_tr);
-	SPDK_CU_ASSERT_FATAL(tr_temp != NULL);
-	TAILQ_REMOVE(&qpair.free_tr, tr_temp, tq_list);
-	tr_temp->req = nvme_allocate_request_null(expected_failure_callback, NULL);
-	SPDK_CU_ASSERT_FATAL(tr_temp->req != NULL);
-
-	tr_temp->req->cmd.opc = SPDK_NVME_OPC_ASYNC_EVENT_REQUEST;
-	tr_temp->req->cmd.cid = tr_temp->cid;
-	TAILQ_INSERT_HEAD(&qpair.outstanding_tr, tr_temp, tq_list);
-
-	nvme_qpair_destroy(&qpair);
-	CU_ASSERT(TAILQ_EMPTY(&qpair.outstanding_tr));
-}
 #endif
 
+static uint64_t g_vtophys_size = 0;
+
+DEFINE_RETURN_MOCK(spdk_vtophys, uint64_t);
+uint64_t
+spdk_vtophys(void *buf, uint64_t *size)
+{
+	if (size) {
+		*size = g_vtophys_size;
+	}
+
+	HANDLE_RETURN_MOCK(spdk_vtophys);
+
+	return (uintptr_t)buf;
+}
+
+DEFINE_STUB(spdk_nvme_ctrlr_get_process, struct spdk_nvme_ctrlr_process *,
+	    (struct spdk_nvme_ctrlr *ctrlr, pid_t pid), NULL);
+DEFINE_STUB(nvme_completion_is_retry, bool, (const struct spdk_nvme_cpl *cpl), false);
+DEFINE_STUB_V(spdk_nvme_qpair_print_command, (struct spdk_nvme_qpair *qpair,
+		struct spdk_nvme_cmd *cmd));
+DEFINE_STUB_V(spdk_nvme_qpair_print_completion, (struct spdk_nvme_qpair *qpair,
+		struct spdk_nvme_cpl *cpl));
+
 static void
 prp_list_prep(struct nvme_tracker *tr, struct nvme_request *req, uint32_t *prp_index)
 {
@@ -799,16 +715,75 @@ test_prp_list_append(void)
 					    (NVME_MAX_PRP_LIST_ENTRIES + 1) * 0x1000, 0x1000) == -EINVAL);
 }
 
-static void test_shadow_doorbell_update(void)
+static void
+test_build_contig_hw_sgl_request(void)
 {
-	bool ret;
+	struct spdk_nvme_qpair qpair = {};
+	struct nvme_request req = {};
+	struct nvme_tracker tr = {};
+	int rc;
 
-	/* nvme_pcie_qpair_need_event(uint16_t event_idx, uint16_t new_idx, uint16_t old) */
-	ret = nvme_pcie_qpair_need_event(10, 15, 14);
-	CU_ASSERT(ret == false);
+	/* Test 1: Payload covered by a single mapping */
+	req.payload_size = 100;
+	req.payload = NVME_PAYLOAD_CONTIG(0, 0);
+	g_vtophys_size = 100;
+	MOCK_SET(spdk_vtophys, 0xDEADBEEF);
 
-	ret = nvme_pcie_qpair_need_event(14, 15, 14);
-	CU_ASSERT(ret == true);
+	rc = nvme_pcie_qpair_build_contig_hw_sgl_request(&qpair, &req, &tr);
+	CU_ASSERT(rc == 0);
+	CU_ASSERT(req.cmd.dptr.sgl1.unkeyed.type == SPDK_NVME_SGL_TYPE_DATA_BLOCK);
+	CU_ASSERT(req.cmd.dptr.sgl1.address == 0xDEADBEEF);
+	CU_ASSERT(req.cmd.dptr.sgl1.unkeyed.length == 100);
+
+	MOCK_CLEAR(spdk_vtophys);
+	g_vtophys_size = 0;
+	memset(&qpair, 0, sizeof(qpair));
+	memset(&req, 0, sizeof(req));
+	memset(&tr, 0, sizeof(tr));
+
+	/* Test 2: Payload covered by a single mapping, but request is at an offset */
+	req.payload_size = 100;
+	req.payload_offset = 50;
+	req.payload = NVME_PAYLOAD_CONTIG(0, 0);
+	g_vtophys_size = 1000;
+	MOCK_SET(spdk_vtophys, 0xDEADBEEF);
+
+	rc = nvme_pcie_qpair_build_contig_hw_sgl_request(&qpair, &req, &tr);
+	CU_ASSERT(rc == 0);
+	CU_ASSERT(req.cmd.dptr.sgl1.unkeyed.type == SPDK_NVME_SGL_TYPE_DATA_BLOCK);
+	CU_ASSERT(req.cmd.dptr.sgl1.address == 0xDEADBEEF);
+	CU_ASSERT(req.cmd.dptr.sgl1.unkeyed.length == 100);
+
+	MOCK_CLEAR(spdk_vtophys);
+	g_vtophys_size = 0;
+	memset(&qpair, 0, sizeof(qpair));
+	memset(&req, 0, sizeof(req));
+	memset(&tr, 0, sizeof(tr));
+
+	/* Test 3: Payload spans two mappings */
+	req.payload_size = 100;
+	req.payload = NVME_PAYLOAD_CONTIG(0, 0);
+	g_vtophys_size = 60;
+	tr.prp_sgl_bus_addr = 0xFF0FF;
+	MOCK_SET(spdk_vtophys, 0xDEADBEEF);
+
+	rc = nvme_pcie_qpair_build_contig_hw_sgl_request(&qpair, &req, &tr);
+	CU_ASSERT(rc == 0);
+	CU_ASSERT(req.cmd.dptr.sgl1.unkeyed.type == SPDK_NVME_SGL_TYPE_LAST_SEGMENT);
+	CU_ASSERT(req.cmd.dptr.sgl1.address == tr.prp_sgl_bus_addr);
+	CU_ASSERT(req.cmd.dptr.sgl1.unkeyed.length == 2 * sizeof(struct spdk_nvme_sgl_descriptor));
+	CU_ASSERT(tr.u.sgl[0].unkeyed.type == SPDK_NVME_SGL_TYPE_DATA_BLOCK);
+	CU_ASSERT(tr.u.sgl[0].unkeyed.length = 60);
+	CU_ASSERT(tr.u.sgl[0].address = 0xDEADBEEF);
+	CU_ASSERT(tr.u.sgl[1].unkeyed.type == SPDK_NVME_SGL_TYPE_DATA_BLOCK);
+	CU_ASSERT(tr.u.sgl[1].unkeyed.length = 40);
+	CU_ASSERT(tr.u.sgl[1].address = 0xDEADBEEF);
+
+	MOCK_CLEAR(spdk_vtophys);
+	g_vtophys_size = 0;
+	memset(&qpair, 0, sizeof(qpair));
+	memset(&req, 0, sizeof(req));
+	memset(&tr, 0, sizeof(tr));
 }
 
 int main(int argc, char **argv)
@@ -826,9 +801,8 @@ int main(int argc, char **argv)
 		return CU_get_error();
 	}
 
-	if (CU_add_test(suite, "prp_list_append", test_prp_list_append) == NULL
-	    || CU_add_test(suite, "shadow_doorbell_update",
-			   test_shadow_doorbell_update) == NULL) {
+	if (CU_add_test(suite, "prp_list_append", test_prp_list_append) == NULL ||
+	    CU_add_test(suite, "build_contig_hw_sgl_request", test_build_contig_hw_sgl_request) == NULL) {
 		CU_cleanup_registry();
 		return CU_get_error();
 	}

test/unit/lib/nvme/nvme_qpair.c/nvme_qpair_ut.c

@@ -243,18 +243,18 @@ static void test_nvme_qpair_process_completions(void)
 	ctrlr.is_removed = false;
 	ctrlr.is_resetting = true;
 	rc = spdk_nvme_qpair_process_completions(&qpair, 0);
-	CU_ASSERT(rc == 0);
+	CU_ASSERT(rc == -ENXIO);
 	CU_ASSERT(g_called_transport_process_completions == false);
 	/* We also need to make sure we didn't abort the requests. */
 	CU_ASSERT(!STAILQ_EMPTY(&qpair.queued_req));
 	CU_ASSERT(g_num_cb_passed == 0);
 	CU_ASSERT(g_num_cb_failed == 0);
 
-	/* The case where we aren't resetting, but are enablign the qpair is the same as above. */
+	/* The case where we aren't resetting, but are enabling the qpair is the same as above. */
 	ctrlr.is_resetting = false;
 	qpair.state = NVME_QPAIR_ENABLING;
 	rc = spdk_nvme_qpair_process_completions(&qpair, 0);
-	CU_ASSERT(rc == 0);
+	CU_ASSERT(rc == -ENXIO);
 	CU_ASSERT(g_called_transport_process_completions == false);
 	CU_ASSERT(!STAILQ_EMPTY(&qpair.queued_req));
 	CU_ASSERT(g_num_cb_passed == 0);

test/unit/lib/nvmf/ctrlr.c/ctrlr_ut.c

@@ -583,19 +583,44 @@ test_connect(void)
 	CU_ASSERT(rsp.connect_rsp.status_code_specific.invalid.ipo == 42);
 	CU_ASSERT(qpair.ctrlr == NULL);
 
-	/* I/O connect to discovery controller keep-alive-timeout should be 0 */
+	/* I/O connect to discovery controller with keep-alive-timeout != 0 */
 	cmd.connect_cmd.qid = 0;
+	cmd.connect_cmd.kato = 120000;
 	memset(&rsp, 0, sizeof(rsp));
 	subsystem.subtype = SPDK_NVMF_SUBTYPE_DISCOVERY;
 	subsystem.state = SPDK_NVMF_SUBSYSTEM_ACTIVE;
 	TAILQ_INSERT_TAIL(&qpair.outstanding, &req, link);
 	rc = spdk_nvmf_ctrlr_connect(&req);
 	poll_threads();
-	CU_ASSERT(rc == SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE);
-	CU_ASSERT(rsp.nvme_cpl.status.sct == SPDK_NVME_SCT_GENERIC);
-	CU_ASSERT(rsp.nvme_cpl.status.sc == SPDK_NVME_SC_INTERNAL_DEVICE_ERROR);
-	CU_ASSERT(qpair.ctrlr == NULL);
+	CU_ASSERT(rc == SPDK_NVMF_REQUEST_EXEC_STATUS_ASYNCHRONOUS);
+	CU_ASSERT(nvme_status_success(&rsp.nvme_cpl.status));
+	CU_ASSERT(qpair.ctrlr != NULL);
+	CU_ASSERT(qpair.ctrlr->keep_alive_poller != NULL);
+	spdk_nvmf_ctrlr_stop_keep_alive_timer(qpair.ctrlr);
+	spdk_bit_array_free(&qpair.ctrlr->qpair_mask);
+	free(qpair.ctrlr);
+	qpair.ctrlr = NULL;
+
+	/* I/O connect to discovery controller with keep-alive-timeout == 0.
+	 *  Then, a fixed timeout value is set to keep-alive-timeout.
+	 */
+	cmd.connect_cmd.kato = 0;
+	memset(&rsp, 0, sizeof(rsp));
+	subsystem.subtype = SPDK_NVMF_SUBTYPE_DISCOVERY;
+	subsystem.state = SPDK_NVMF_SUBSYSTEM_ACTIVE;
+	TAILQ_INSERT_TAIL(&qpair.outstanding, &req, link);
+	rc = spdk_nvmf_ctrlr_connect(&req);
+	poll_threads();
+	CU_ASSERT(rc == SPDK_NVMF_REQUEST_EXEC_STATUS_ASYNCHRONOUS);
+	CU_ASSERT(nvme_status_success(&rsp.nvme_cpl.status));
+	CU_ASSERT(qpair.ctrlr != NULL);
+	CU_ASSERT(qpair.ctrlr->keep_alive_poller != NULL);
+	spdk_nvmf_ctrlr_stop_keep_alive_timer(qpair.ctrlr);
+	spdk_bit_array_free(&qpair.ctrlr->qpair_mask);
+	free(qpair.ctrlr);
+	qpair.ctrlr = NULL;
 	cmd.connect_cmd.qid = 1;
+	cmd.connect_cmd.kato = 120000;
 	subsystem.subtype = SPDK_NVMF_SUBTYPE_NVME;
 
 	/* I/O connect to disabled controller */