env_dpdk/pci: fix check on 20.11 EAL API change

RTE_DEV_ALLOWED is an enum and has no associated define, hence checking
for its presence will always be false.
We could test for RTE_DEV_WHITELISTED define, but this macro added for
deprecation warning will be dropped in the future.
Switch to a check on DPDK version.

Fixes: 10ed0eb755 ("env_dpdk/pci: adapt to 20.11 EAL changes")
Signed-off-by: David Marchand <david.marchand@redhat.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5165 (master)

(cherry picked from commit 2e9cd9d7b0)
Change-Id: I75270977b580065b36c753266cbaa5fb73f99eb1
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5175
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

.github/mistaken-pull-closer.yml

@@ -1,10 +0,0 @@
-filters:
-  - true
-
-commentBody: |
-  Thanks for your contribution!  Unfortunately, we don't use GitHub pull
-  requests to manage code contributions to this repository.  Instead, please
-  see https://spdk.io/development which provides instructions on how to
-  submit patches to the SPDK Gerrit instance.
-
-addLabel: false

.gitignore

@@ -2,17 +2,12 @@
 *.a
 *.cmd
 *.d
-*.dll
-*.exe
 *.gcda
 *.gcno
 *.kdev4
 *.ko
-*.lib
 *.log
 *.o
-*.obj
-*.pdb
 *.pyc
 *.so
 *.so.*

.gitmodules

@@ -1,6 +1,6 @@
 [submodule "dpdk"]
 	path = dpdk
-	url = https://git.quacker.org/d/numam-dpdk.git
+	url = https://github.com/spdk/dpdk.git
 [submodule "intel-ipsec-mb"]
 	path = intel-ipsec-mb
 	url = https://github.com/spdk/intel-ipsec-mb.git
@@ -10,6 +10,3 @@
 [submodule "ocf"]
 	path = ocf
 	url = https://github.com/Open-CAS/ocf.git
-[submodule "libvfio-user"]
-	path = libvfio-user
-	url = https://github.com/nutanix/libvfio-user.git

CHANGELOG.md

@@ -1,396 +1,8 @@
 # Changelog
 
-## v21.04.1: (Upcoming Release)
+## v20.10.1: (Upcoming Release)
 
-## v21.04: ZNS NVMe bdev, PMR, ADQ initiator, RPM
-
-### accel
-
-Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and
-`spdk_accel_batch_prep_crc32cv` are added in order to provide the
-chained accelerated CRC32 computation support.
-
-### bdev
-
-For `bdev_ocssd_create` RPC, the optional parameter `range` was removed.
-Only one OCSSD bdev can be created for one OCSSD namespace.
-
-Removed the `spdk_bdev_open` from bdev library API.
-Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API.
-Removed the `config_text` function for bdev modules to report legacy config.
-
-Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev.
-
-Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size.
-
-### bdev_nvme
-
-Added support for zoned namespaces.
-
-### blobstore
-
-Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API.
-
-### env
-
-Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to
-the allowed list after the application has started.
-
-Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`.
-
-Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in
-nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`,
-`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`.
-
-### event
-
-Removed the `config_file`, `max_delay_us`, `pci_whitelist`
-and `pci_blacklist` members of struct `spdk_app_opts`.
-
-### idxd
-
-A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library
-can know whether they need to rebalance the flow control for the channel
-that was just added/removed.  This is based on how the low level library
-shares devices amongst channels.
-
-The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this
-is now tracked in the library.  The app makes use the new API above to
-determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan`
-as well.
-
-The API `spdk_idxd_put_channel` now returns the rebalance state for the
-underlying device.
-
-### iscsi
-
-A security vulnerability has been identified and fixed in the SPDK iSCSI target.
-A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference
-and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target
-are recommended to update. All SPDK versions <= v21.01 are affected.
-
-### net
-
-The net library is deprecated and will be removed in the 21.07 release.
-
-### nvme
-
-Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities.
-
-Added an accelerated table pointer in `spdk_nvme_poll_group`
-which can be used to provide the accelerated functions by users with
-hardware engine, such as crc32c accelerated function.
-
-Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`,
-`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`.
-
-Added NVMe transport operations to enable, disable, map and unmap the PMR.
-
-Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group`
-function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most
-suitable polling group.
-
-Added OPTPERF and namespace optimal performance fields to nvme_spec.h.
-
-Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which
-hot-inserted SSDs should be probed. This is useful for use cases where multiple
-independent SPDK processes are running on one node. The filter function can
-then be implemented in these processes to decide which SSDs to probe based on
-the new SSD's PCI address.
-
-New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats`
-were added. These functions allow to get transport statistics per NVME poll group.
-
-Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases.
-
-Added support for vector variant of ZNS zone append commands with new API
-`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`.
-
-Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API,
-to display maximum number of open and active zones of the given namespace.
-
-Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of
-sectors.
-
-Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair.
-
-### nvmf
-
-Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API.
-
-Added new APIs:
-
-- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`).
-- `poll_group_dump_stat` (transport op in `nvmf_transport.h`).
-
-The following APIs have been deprecated and will be removed in SPDK 21.07:
-
-- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`),
-- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`),
-- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`),
-- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`),
-- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`),
-- `poll_group_get_stat` (transport op in `nvmf_transport.h`),
-- `poll_group_free_stat` (transport op in `nvmf_transport.h`).
-
-See header files for details.
-
-The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport,
-which is the same as the type for defined transports and more informative for a custom transport.
-
-Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change
-state of TCP header and data digest.
-
-Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ)
-for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP.
-
-### ocf
-
-Updated OCF submodule to v20.12.2
-
-Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev.
-
-### opal
-
-Removed the `spdk_opal_supported` API.
-
-### raid
-
-For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed.
-
-### rpc
-
-New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics
-of nvme poll groups.
-
-Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07,
-use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead.
-Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07,
-use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead.
-
-### rpm
-
-Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the
-SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07.
-See [RPM documentation](https://spdk.io/doc/rpm.html) for more details.
-
-### sock
-
-The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from
-bool to int. We can use RPC to configure different value of `enable_placement_id`.
-Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize
-CPU cache locality, enabled by setting enable_placement_id=2.
-
-A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow
-sockets to be marked using the SO_MARK socket option as a hint for which hardware
-queue they should be associated with. This mode leverages that by setting the same
-value for all sockets within a poll group.
-
-New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added
-to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server
-and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server);
-`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter
-`enable_zerocopy_send`  enables or disables zero copy send for both server and client sockets.
-
-### thread
-
-A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified
-I/O channel.
-
-Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into
-interrupt mode or back to poll mode. It is valid only when thread interrupt facility is
-enabled by `spdk_interrupt_mode_enable`.
-
-Added `spdk_poller_register_interrupt` function to mark that the poller is capable of
-entering interrupt mode. Callback function will be called when the poller must transition
-into or out of interrupt mode.
-
-### virtio
-
-Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device.
-
-## v21.01:
-
-### bdev
-
-An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the
-ABI compatibility issue between different SPDK version. And also add `opts_size`
-parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and
-`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine
-the small and large buffer pool size of the whole bdev module.
-
-A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of
-examine process. Along with corresponding `bdev_wait_for_examine` RPC, which
-is now always called during `spdk_bdev_subsystem_config_json` making sure
-every bdev is ready to be used.
-
-A new API `spdk_bdev_io_get_aio_status` was added for getting the status of
-bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function
-and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete
-a bdev_io with Linux AIO errno.
-
-A new API `spdk_bdev_get_module_name` was added for getting block device module name.
-Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for
-the block device opened by the specified descriptor.
-
-Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for
-bdev modules to specify splitting requirements.
-
-### blob
-
-An `opts_size` element was added in the `spdk_bs_opts` structure to solve the
-ABI compatibility issue between different SPDK version. And also add `opts_size`
-parameter in `spdk_bs_opts_init` function.
-
-An `opts_size` element was added in the `spdk_blob_opts` structure to solve the
-ABI compatibility issue between different SPDK version. And also add `opts_size`
-parameter in `spdk_blob_opts_init` function.
-
-An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the
-ABI compatibility issue between different SPDK version. And also add `opts_size`
-parameter in `spdk_blob_open_opts_init` function.
-
-### build
-
-SPDK now generates pkg-config files to simplify the process of determining which
-libraries must be linked into an SPDK application.
-
-### dpdk
-
-Updated DPDK submodule to DPDK 20.11.
-
-Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver
-was moved to separate dpdk-kmods repository. If required, users need to refer to
-dpdk-kmods repository to build the driver.
-
-### env
-
-The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`
-have been deprecated.  The new members are named `pci_allowed`, `pci_blocked` and
-`main_core` respectively.
-
-An `opts_size` element was added in the `spdk_app_opts` structure
-to solve the ABI compatibility issue between different SPDK version. An `opts_size`
-parameter is added into `spdk_app_opts_init` function.
-
-### event
-
-The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been
-deprecated.  The new members are named `pci_allowed` and `pci_blocked` respectively.
-
-The `--pci-blacklist` command line option has been deprecated, replaced with
-`--pci-blocked`.
-
-The `--pci-whitelist/-W` command line options have been deprecated, replaced with
-`--pci-allowed/-A`.
-
-Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency
-using dpdk_governor and turns idle reactor cores to interrupt mode. Please see
-[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details.
-
-## ioat
-
-The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC.
-IOAT PCI functions can still be allowed or denied using SPDK application command
-line options.
-
-### isa-l
-
-Updated ISA-L submodule to v2.30.0.
-
-### json
-
-A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for
-writing json bool results into one function.
-Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you
-can specify the `keep_alive_timeout` before creating NVMe bdev.
-
-### nbd
-
-Change the return type of function `spdk_nbd_stop` from void to int. And update the
-`spdk_nbd_fini` with two parameters to make its behavior from sync to async.
-
-### nvme
-
-Directives support was added to the NVMe driver.
-
-Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send`
-are added for Directive Send and Directive Receive command, respectively.
-
-Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe
-subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet.
-
-Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of
-dwords.
-
-Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe
-driver running with NVMe over vfio-user target.
-
-Added the vfio-user custom transport implementation in NVMe driver which can connect
-to NVMe over vfio-user target via vfio-user transport.
-
-Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override
-`remove_cb` and `remove_ctx` specified when the controller was probed.
-
-Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and
-`spdk_nvme_zns_zone_append_with_md`.
-
-### nvmf
-
-`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver
-to the latest is required.
-
-FC transport supported primary tagging and VMID.
-
-Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API,
-`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`.
-Hence updating Broadcom FC LLD driver to the latest is required.
-
-The functions `destroy` and `qpair_fini` in the transport interface now accept a
-`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous.
-
-The SPDK NVMe-oF target now supports async event notification for discovery log changes.
-This allows the initiator to create persistent connection to discovery controller and
-be notified of any discovery log changes.
-
-An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure
-to solve the ABI compatiblity issue between different SPDK version. And also add
-`opts_size` parameter in `spdk_nvmf_transport_opts_init` function.
-
-Added a new custom vfio-user transport implementation in NVMe-oF which can provide
-emulated NVMe devices to QEMU and SPDK NVMe driver.
-
-Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow
-passing generic options to specific transport. This functionality replaces one provided
-with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC
-that allowed transport specific options within subsystem.
-
-The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional.
-
-Pausing a subsystem now only pauses all admin queues. This allows for the
-management of hosts and listeners, plus the addition of namespaces without a
-full subsystem pause. Additionally, the target now allows for pausing
-individual namespaces within a subsystem. To remove a namespace from a
-subsystem, only the specific namespace must be paused. I/O will continue to
-other namespaces while these operations execute.
-
-### rpc
-
-An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`,
-and an new RPC `iscsi_start_portal_group` was added. They will be used not to start
-listening on portals for a portal group until all associated target nodes are created
-at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI
-target application restarts.
-
-Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added
-into `bdev_set_options` function.
-
-### vhost
-
-Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC
-enabling packed ring live recovery.
-
-## v20.10:
+## v20.10: NVMe-oF multipath, NVMe ZNS, iSCSI login redirection
 
 ### accel
 
@@ -558,15 +170,14 @@ New optional parameters, `enable_placement_id` and `enable_quickack` were added
 A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly.
 It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`.
 
-New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC
-'nvmf_create_transport'.
+Add optional 'no_wr_batching' parameter to 'nvmf_create_transport' RPC method.
 
 New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have
 been added, and a new optional parameter `private` has been added to the RPC
 `iscsi_create_portal_group` to support iSCSI login redirection feature.
 
 A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and
-a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting.
+a new RPC `nvmf_subsystem_listner_set_ana_state` was added for ANA reporting.
 
 New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to
 retrieve configuration of the NVMe-oF subsystem.
@@ -2639,7 +2250,7 @@ may be set via the configure `--prefix` option.  Example: `make install prefix=/
 
 ### RPC
 
-A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock.
+A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/run/spdk.sock.
 A -r option command line option has been added to enable an alternative UNIX domain socket location,
 or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260).  The Rpc configuration file
 section is now deprecated and will be removed in the v18.04 release.

CONFIG

@@ -76,9 +76,6 @@ CONFIG_UNIT_TESTS=y
 # Build examples
 CONFIG_EXAMPLES=y
 
-# Build with Control-flow Enforcement Technology (CET)
-CONFIG_CET=n
-
 # Directory that contains the desired SPDK environment library.
 # By default, this is implemented using DPDK.
 CONFIG_ENV=
@@ -87,9 +84,6 @@ CONFIG_ENV=
 # installation.
 CONFIG_DPDK_DIR=
 
-# This directory should contain 'include' and 'lib' directories for WPDK.
-CONFIG_WPDK_DIR=
-
 # Build SPDK FIO plugin. Requires CONFIG_FIO_SOURCE_DIR set to a valid
 # fio source code directory.
 CONFIG_FIO_PLUGIN=n
@@ -123,10 +117,6 @@ CONFIG_VHOST=y
 # Build vhost initiator (Virtio) driver.
 CONFIG_VIRTIO=y
 
-# Build custom vfio-user transport for NVMf target and NVMe initiator.
-CONFIG_VFIO_USER=n
-CONFIG_VFIO_USER_DIR=
-
 # Build with PMDK backends
 CONFIG_PMDK=n
 CONFIG_PMDK_DIR=
@@ -147,6 +137,9 @@ CONFIG_SHARED=n
 CONFIG_VTUNE=n
 CONFIG_VTUNE_DIR=
 
+# Build the dpdk igb_uio driver
+CONFIG_IGB_UIO_DRIVER=n
+
 # Build Intel IPSEC_MB library
 CONFIG_IPSEC_MB=n
 

LICENSE

@@ -1,16 +1,3 @@
-The SPDK repo contains multiple git submodules each with its own
-license info. Unless otherwise noted all other code in this repo
-is BSD as stated below.
-
-Submodule license info:
-dpdk: see dpdk/license
-intel-ipsec-mb: see intel-ipsec-mb/LICENSE
-isa-l: see isa-l/LICENSE
-libvfio-user: see libvfio-user/LICENSE
-ocf: see ocf/LICENSE
-
-The rest of the SPDK repo:
-
 BSD LICENSE
 
 Copyright (c) Intel Corporation.

Makefile

@@ -45,7 +45,6 @@ DIRS-$(CONFIG_EXAMPLES) += examples
 DIRS-y += test
 DIRS-$(CONFIG_IPSEC_MB) += ipsecbuild
 DIRS-$(CONFIG_ISAL) += isalbuild
-DIRS-$(CONFIG_VFIO_USER) += vfiouserbuild
 
 .PHONY: all clean $(DIRS-y) include/spdk/config.h mk/config.mk \
 	cc_version cxx_version .libs_only_other .ldflags ldflags install \
@@ -63,13 +62,6 @@ endif
 endif
 endif
 
-ifeq ($(OS),Windows)
-ifeq ($(CURDIR)/wpdk/build,$(CONFIG_WPDK_DIR))
-WPDK = wpdk
-DIRS-y += wpdk
-endif
-endif
-
 ifeq ($(CONFIG_SHARED),y)
 LIB = shared_lib
 else
@@ -86,11 +78,6 @@ LIB += isalbuild
 DPDK_DEPS += isalbuild
 endif
 
-ifeq ($(CONFIG_VFIO_USER),y)
-VFIOUSERBUILD = vfiouserbuild
-LIB += vfiouserbuild
-endif
-
 all: mk/cc.mk $(DIRS-y)
 clean: $(DIRS-y)
 	$(Q)rm -f include/spdk/config.h
@@ -98,7 +85,6 @@ clean: $(DIRS-y)
 	$(Q)rm -rf build/fio
 	$(Q)rm -rf build/examples
 	$(Q)rm -rf build/include
-	$(Q)rm -rf build/lib/pkgconfig
 	$(Q)find build/lib ! -name .gitignore -type f -delete
 
 install: all
@@ -108,11 +94,10 @@ uninstall: $(DIRS-y)
 	$(Q)echo "Uninstalled spdk"
 
 ifneq ($(SKIP_DPDK_BUILD),1)
-dpdkdeps $(DPDK_DEPS): $(WPDK)
-dpdkbuild: $(WPDK) $(DPDK_DEPS)
+dpdkbuild: $(DPDK_DEPS)
 endif
 
-lib: $(WPDK) $(DPDKBUILD) $(VFIOUSERBUILD)
+lib: $(DPDKBUILD)
 module: lib
 shared_lib: module
 app: $(LIB)
@@ -128,7 +113,7 @@ mk/cc.mk:
 	false
 
 build_dir: mk/cc.mk
-	$(Q)mkdir -p build/lib/pkgconfig/tmp
+	$(Q)mkdir -p build/lib
 	$(Q)mkdir -p build/bin
 	$(Q)mkdir -p build/fio
 	$(Q)mkdir -p build/examples

app/Makefile

@@ -41,10 +41,7 @@ DIRS-y += iscsi_top
 DIRS-y += iscsi_tgt
 DIRS-y += spdk_tgt
 DIRS-y += spdk_lspci
-ifneq ($(OS),Windows)
-# TODO - currently disabled on Windows due to lack of support for curses
 DIRS-y += spdk_top
-endif
 ifeq ($(OS),Linux)
 DIRS-$(CONFIG_VHOST) += vhost
 DIRS-y += spdk_dd

app/iscsi_tgt/Makefile

@@ -43,14 +43,17 @@ CFLAGS += -I$(SPDK_ROOT_DIR)/lib
 
 C_SRCS := iscsi_tgt.c
 
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_iscsi event_net
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM) event_iscsi event_net event_scsi event
+SPDK_LIB_LIST += jsonrpc json rpc bdev iscsi scsi accel trace conf
+SPDK_LIB_LIST += thread util log net sock notify
 
 ifeq ($(SPDK_ROOT_DIR)/lib/env_dpdk,$(CONFIG_ENV))
 SPDK_LIB_LIST += env_dpdk_rpc
 endif
 
 ifeq ($(OS),Linux)
-SPDK_LIB_LIST += event_nbd
+SPDK_LIB_LIST += event_nbd nbd
 endif
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

app/iscsi_tgt/iscsi_tgt.c

@@ -75,7 +75,7 @@ main(int argc, char **argv)
 	int rc;
 	struct spdk_app_opts opts = {};
 
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "iscsi";
 	if ((rc = spdk_app_parse_args(argc, argv, &opts, "b", NULL,
 				      iscsi_parse_arg, iscsi_usage)) !=

app/iscsi_top/Makefile

@@ -37,7 +37,7 @@ include $(SPDK_ROOT_DIR)/mk/spdk.modules.mk
 
 APP = iscsi_top
 
-SPDK_LIB_LIST = rpc
+SPDK_LIB_LIST = jsonrpc json rpc log util
 
 CFLAGS += -I$(SPDK_ROOT_DIR)/lib
 

app/nvmf_tgt/Makefile

@@ -39,14 +39,24 @@ APP = nvmf_tgt
 
 C_SRCS := nvmf_main.c
 
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_nvmf
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM) event_nvmf event_net
+SPDK_LIB_LIST += nvmf event log trace conf thread util bdev accel rpc jsonrpc json net sock
+SPDK_LIB_LIST += notify
 
 ifeq ($(SPDK_ROOT_DIR)/lib/env_dpdk,$(CONFIG_ENV))
 SPDK_LIB_LIST += env_dpdk_rpc
 endif
 
 ifeq ($(OS),Linux)
-SPDK_LIB_LIST += event_nbd
+SPDK_LIB_LIST += event_nbd nbd
+endif
+
+ifeq ($(CONFIG_FC),y)
+ifneq ($(strip $(CONFIG_FC_PATH)),)
+SYS_LIBS += -L$(CONFIG_FC_PATH)
+endif
+SYS_LIBS += -lufc
 endif
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

app/nvmf_tgt/nvmf_main.c

@@ -63,7 +63,7 @@ main(int argc, char **argv)
 	struct spdk_app_opts opts = {};
 
 	/* default value in opts */
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "nvmf";
 	if ((rc = spdk_app_parse_args(argc, argv, &opts, "", NULL,
 				      nvmf_parse_arg, nvmf_usage)) !=

app/spdk_dd/Makefile

@@ -39,6 +39,9 @@ APP = spdk_dd
 
 C_SRCS := spdk_dd.c
 
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_bdev
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += event_sock event_bdev event_accel event_vmd
+SPDK_LIB_LIST += bdev accel event thread util conf trace \
+		log jsonrpc json rpc sock notify
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

app/spdk_dd/spdk_dd.c

@@ -37,7 +37,6 @@
 #include "spdk/event.h"
 #include "spdk/fd.h"
 #include "spdk/string.h"
-#include "spdk/util.h"
 #include "spdk/vmd.h"
 
 #include <libaio.h>
@@ -1105,7 +1104,7 @@ main(int argc, char **argv)
 	struct spdk_app_opts opts = {};
 	int rc = 1;
 
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "spdk_dd";
 	opts.reactor_mask = "0x1";
 	opts.shutdown_cb = dd_finish;

app/spdk_lspci/Makefile

@@ -33,12 +33,19 @@
 
 SPDK_ROOT_DIR := $(abspath $(CURDIR)/../..)
 include $(SPDK_ROOT_DIR)/mk/spdk.common.mk
-include $(SPDK_ROOT_DIR)/mk/spdk.modules.mk
 
 APP = spdk_lspci
 
 C_SRCS := spdk_lspci.c
 
-SPDK_LIB_LIST = $(SOCK_MODULES_LIST) nvme vmd
+SPDK_LIB_LIST = $(SOCK_MODULES_LIST)
+SPDK_LIB_LIST += nvme thread util log sock vmd jsonrpc json rpc
+
+ifeq ($(CONFIG_RDMA),y)
+SPDK_LIB_LIST += rdma
+ifeq ($(CONFIG_RDMA_PROV),mlx5_dv)
+SYS_LIBS += -lmlx5
+endif
+endif
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

app/spdk_tgt/Makefile

@@ -41,17 +41,29 @@ C_SRCS := spdk_tgt.c
 
 SPDK_LIB_LIST = $(ALL_MODULES_LIST)
 
-SPDK_LIB_LIST += event_iscsi event_nvmf
+ifeq ($(OS),Linux)
+ifeq ($(CONFIG_VHOST),y)
+SPDK_LIB_LIST += vhost event_vhost
+endif
+endif
+
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM) event_iscsi event_net event_scsi event_nvmf event
+SPDK_LIB_LIST += nvmf trace log conf thread util bdev iscsi scsi accel rpc jsonrpc json
+SPDK_LIB_LIST += net sock notify
 
 ifeq ($(SPDK_ROOT_DIR)/lib/env_dpdk,$(CONFIG_ENV))
 SPDK_LIB_LIST += env_dpdk_rpc
 endif
 
 ifeq ($(OS),Linux)
-SPDK_LIB_LIST += event_nbd
-ifeq ($(CONFIG_VHOST),y)
-SPDK_LIB_LIST += event_vhost
+SPDK_LIB_LIST += event_nbd nbd
 endif
+
+ifeq ($(CONFIG_FC),y)
+ifneq ($(strip $(CONFIG_FC_PATH)),)
+SYS_LIBS += -L$(CONFIG_FC_PATH)
+endif
+SYS_LIBS += -lufc
 endif
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

app/spdk_tgt/spdk_tgt.c

@@ -109,7 +109,7 @@ main(int argc, char **argv)
 	struct spdk_app_opts opts = {};
 	int rc;
 
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "spdk_tgt";
 	if ((rc = spdk_app_parse_args(argc, argv, &opts, g_spdk_tgt_get_opts_string,
 				      NULL, spdk_tgt_parse_arg, spdk_tgt_usage)) !=

app/spdk_top/Makefile

@@ -38,7 +38,7 @@ APP = spdk_top
 
 C_SRCS := spdk_top.c
 
-SPDK_LIB_LIST = rpc
+SPDK_LIB_LIST = jsonrpc json rpc log util
 LIBS=-lncurses -lpanel -lmenu
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

app/trace_record/trace_record.c

@@ -613,8 +613,6 @@ int main(int argc, char **argv)
 			file_name = optarg;
 			break;
 		case 'h':
-			usage();
-			exit(EXIT_SUCCESS);
 		default:
 			usage();
 			exit(1);

app/vhost/Makefile

@@ -39,7 +39,13 @@ APP = vhost
 
 C_SRCS := vhost.c
 
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_vhost event_nbd
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += vhost event_vhost
+
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM) event_net event_scsi event
+SPDK_LIB_LIST += jsonrpc json rpc bdev scsi accel trace conf
+SPDK_LIB_LIST += thread util log
+SPDK_LIB_LIST += event_nbd nbd net sock notify
 
 ifeq ($(SPDK_ROOT_DIR)/lib/env_dpdk,$(CONFIG_ENV))
 SPDK_LIB_LIST += env_dpdk_rpc

app/vhost/vhost.c

@@ -88,7 +88,7 @@ main(int argc, char *argv[])
 	struct spdk_app_opts opts = {};
 	int rc;
 
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "vhost";
 
 	if ((rc = spdk_app_parse_args(argc, argv, &opts, "f:S:", NULL,

autobuild.sh

@@ -12,7 +12,6 @@ rootdir=$(readlink -f $(dirname $0))
 
 source "$1"
 source "$rootdir/test/common/autotest_common.sh"
-source "$rootdir/scripts/common.sh"
 
 out=$output_dir
 if [ -n "$SPDK_TEST_NATIVE_DPDK" ]; then
@@ -63,15 +62,12 @@ function build_native_dpdk() {
 
 	rm -rf "$external_dpdk_base_dir"
 	git clone --branch $SPDK_TEST_NATIVE_DPDK --depth 1 http://dpdk.org/git/dpdk "$external_dpdk_base_dir"
-	git -C "$external_dpdk_base_dir" log --oneline -n 5
 
 	dpdk_cflags="-fPIC -g -Werror -fcommon"
 	dpdk_ldflags=""
 
 	# the drivers we use
-	# net/i40e driver is not really needed by us, but it's built as a workaround
-	# for DPDK issue: https://bugs.dpdk.org/show_bug.cgi?id=576
-	DPDK_DRIVERS=("bus" "bus/pci" "bus/vdev" "mempool/ring" "net/i40e" "net/i40e/base")
+	DPDK_DRIVERS=("bus" "bus/pci" "bus/vdev" "mempool/ring")
 	# all possible DPDK drivers
 	DPDK_ALL_DRIVERS=($(find "$external_dpdk_base_dir/drivers" -mindepth 1 -type d | sed -n "s#^$external_dpdk_base_dir/drivers/##p"))
 
@@ -119,12 +115,9 @@ function build_native_dpdk() {
 		if grep "20.08.0" $external_dpdk_base_dir/VERSION; then
 			wget https://github.com/spdk/dpdk/commit/64f1ced13f974e8b3d46b87c361a09eca68126f9.patch -O dpdk-pci.patch
 			wget https://github.com/spdk/dpdk/commit/c2c273d5c8fbf673623b427f8f4ab5af5ddf0e08.patch -O dpdk-qat.patch
-		elif grep "20.11\|21.02" $external_dpdk_base_dir/VERSION; then
+		else
 			wget https://github.com/karlatec/dpdk/commit/3219c0cfc38803aec10c809dde16e013b370bda9.patch -O dpdk-pci.patch
 			wget https://github.com/karlatec/dpdk/commit/adf8f7638de29bc4bf9ba3faf12bbdae73acda0c.patch -O dpdk-qat.patch
-		else
-			wget https://github.com/karlatec/dpdk/commit/f95e331be3a1f856b816948990dd2afc67ea4020.patch -O dpdk-pci.patch
-			wget https://github.com/karlatec/dpdk/commit/6fd2fa906ffdcee04e6ce5da40e61cb841be9827.patch -O dpdk-qat.patch
 		fi
 		git config --local user.name "spdk"
 		git config --local user.email "nomail@all.com"
@@ -226,15 +219,11 @@ function test_make_uninstall() {
 }
 
 function build_doc() {
-	local doxygenv
-	doxygenv=$(doxygen --version)
-
 	$MAKE -C "$rootdir"/doc --no-print-directory $MAKEFLAGS &> "$out"/doxygen.log
 	if [ -s "$out"/doxygen.log ]; then
 		cat "$out"/doxygen.log
 		echo "Doxygen errors found!"
-		eq "$doxygenv" 1.8.20 || exit 1
-		echo "Doxygen $doxygenv detected, all warnings are potentially false positives, continuing the test"
+		exit 1
 	fi
 	if hash pdflatex 2> /dev/null; then
 		$MAKE -C "$rootdir"/doc/output/latex --no-print-directory $MAKEFLAGS &>> "$out"/doxygen.log
@@ -246,8 +235,7 @@ function build_doc() {
 	fi
 	$MAKE -C "$rootdir"/doc --no-print-directory $MAKEFLAGS clean &>> "$out"/doxygen.log
 	if [ -s "$out"/doxygen.log ]; then
-		# Save the log as an artifact in case we are working with potentially broken version
-		eq "$doxygenv" 1.8.20 || rm "$out"/doxygen.log
+		rm "$out"/doxygen.log
 	fi
 	rm -rf "$rootdir"/doc/output
 }

autopackage.sh

@@ -13,37 +13,6 @@ source "$1"
 rootdir=$(readlink -f $(dirname $0))
 source "$rootdir/test/common/autotest_common.sh"
 
-function build_rpms() (
-	local version rpms
-
-	# Make sure linker will not attempt to look under DPDK's repo dir to get the libs
-	unset -v LD_LIBRARY_PATH
-
-	install_uninstall_rpms() {
-		rpms=("$HOME/rpmbuild/RPMS/x86_64/"spdk{,-devel,{,-dpdk}-libs}-$version-1.x86_64.rpm)
-
-		sudo rpm -i "${rpms[@]}"
-		rpms=("${rpms[@]##*/}") rpms=("${rpms[@]%.rpm}")
-		# Check if we can find one of the apps in the PATH now and verify if it doesn't miss
-		# any libs.
-		LIST_LIBS=yes "$rootdir/rpmbuild/rpm-deps.sh" "${SPDK_APP[@]##*/}"
-		sudo rpm -e "${rpms[@]}"
-	}
-
-	build_rpm() {
-		MAKEFLAGS="$MAKEFLAGS" SPDK_VERSION="$version" DEPS=no "$rootdir/rpmbuild/rpm.sh" "$@"
-		install_uninstall_rpms
-	}
-
-	version="test_shared"
-	run_test "build_shared_rpm" build_rpm --with-shared
-
-	if [[ -n $SPDK_TEST_NATIVE_DPDK ]]; then
-		version="test_shared_native_dpdk"
-		run_test "build_shared_native_dpdk_rpm" build_rpm --with-shared --with-dpdk="$SPDK_RUN_EXTERNAL_DPDK"
-	fi
-)
-
 out=$PWD
 
 MAKEFLAGS=${MAKEFLAGS:--j16}
@@ -59,12 +28,7 @@ if [ $(git status --porcelain --ignore-submodules | wc -l) -ne 0 ]; then
 fi
 timing_exit porcelain_check
 
-if [[ $SPDK_TEST_RELEASE_BUILD -eq 1 ]]; then
-	run_test "build_rpms" build_rpms
-	$MAKE clean
-fi
-
-if [[ $RUN_NIGHTLY -eq 0 ]]; then
+if [[ $RUN_NIGHTLY -eq 0 && $SPDK_TEST_RELEASE_BUILD -eq 0 ]]; then
 	timing_finish
 	exit 0
 fi

autorun.sh

@@ -4,8 +4,7 @@ set -e
 
 rootdir=$(readlink -f $(dirname $0))
 
-default_conf=~/autorun-spdk.conf
-conf=${1:-${default_conf}}
+conf=~/autorun-spdk.conf
 
 # If the configuration of tests is not provided, no tests will be carried out.
 if [[ ! -f $conf ]]; then

autotest.sh

@@ -30,10 +30,8 @@ fi
 
 if [ $(uname -s) = Linux ]; then
 	old_core_pattern=$(< /proc/sys/kernel/core_pattern)
-	mkdir -p "$output_dir/coredumps"
 	# set core_pattern to a known value to avoid ABRT, systemd-coredump, etc.
-	echo "|$rootdir/scripts/core-collector.sh %P %s %t $output_dir/coredumps" > /proc/sys/kernel/core_pattern
-	echo 2 > /proc/sys/kernel/core_pipe_limit
+	echo "core" > /proc/sys/kernel/core_pattern
 
 	# Make sure that the hugepage state for our VM is fresh so we don't fail
 	# hugepage allocation. Allow time for this action to complete.
@@ -60,13 +58,11 @@ src=$(readlink -f $(dirname $0))
 out=$output_dir
 cd $src
 
+./scripts/setup.sh status
+
 freebsd_update_contigmem_mod
 
-# lcov takes considerable time to process clang coverage.
-# Disabling lcov allow us to do this.
-# More information: https://github.com/spdk/spdk/issues/1693
-CC_TYPE=$(grep CC_TYPE mk/cc.mk)
-if hash lcov && ! [[ "$CC_TYPE" == *"clang"* ]]; then
+if hash lcov; then
 	# setup output dir for unittest.sh
 	export UT_COVERAGE=$out/ut_coverage
 	export LCOV_OPTS="
@@ -94,7 +90,7 @@ rm -f /var/tmp/spdk*.sock
 
 if [ $(uname -s) = Linux ]; then
 	# OCSSD devices drivers don't support IO issues by kernel so
-	# detect OCSSD devices and block them (unbind from any driver).
+	# detect OCSSD devices and blacklist them (unbind from any driver).
 	# If test scripts want to use this device it needs to do this explicitly.
 	#
 	# If some OCSSD device is bound to other driver than nvme we won't be able to
@@ -104,32 +100,29 @@ if [ $(uname -s) = Linux ]; then
 		# Send Open Channel 2.0 Geometry opcode "0xe2" - not supported by NVMe device.
 		if nvme admin-passthru $dev --namespace-id=1 --data-len=4096 --opcode=0xe2 --read > /dev/null; then
 			bdf="$(basename $(readlink -e /sys/class/nvme/${dev#/dev/}/device))"
-			echo "INFO: blocking OCSSD device: $dev ($bdf)"
-			PCI_BLOCKED+=" $bdf"
+			echo "INFO: blacklisting OCSSD device: $dev ($bdf)"
+			PCI_BLACKLIST+=" $bdf"
 			OCSSD_PCI_DEVICES+=" $bdf"
 		fi
 	done < <(find /dev -maxdepth 1 -regex '/dev/nvme[0-9]+' -print0)
 
 	export OCSSD_PCI_DEVICES
 
-	# Now, bind blocked devices to pci-stub module. This will prevent
+	# Now, bind blacklisted devices to pci-stub module. This will prevent
 	# automatic grabbing these devices when we add device/vendor ID to
 	# proper driver.
-	if [[ -n "$PCI_BLOCKED" ]]; then
+	if [[ -n "$PCI_BLACKLIST" ]]; then
 		# shellcheck disable=SC2097,SC2098
-		PCI_ALLOWED="$PCI_BLOCKED" \
-			PCI_BLOCKED="" \
+		PCI_WHITELIST="$PCI_BLACKLIST" \
+			PCI_BLACKLIST="" \
 			DRIVER_OVERRIDE="pci-stub" \
 			./scripts/setup.sh
 
-		# Export our blocked list so it will take effect during next setup.sh
-		export PCI_BLOCKED
+		# Export our blacklist so it will take effect during next setup.sh
+		export PCI_BLACKLIST
 	fi
-	run_test "setup.sh" "$rootdir/test/setup/test-setup.sh"
 fi
 
-./scripts/setup.sh status
-
 if [[ $(uname -s) == Linux ]]; then
 	# Revert NVMe namespaces to default state
 	nvme_namespace_revert
@@ -151,13 +144,12 @@ timing_enter afterboot
 ./scripts/setup.sh
 timing_exit afterboot
 
+timing_enter nvmf_setup
+rdma_device_init
+timing_exit nvmf_setup
+
 if [[ $SPDK_TEST_CRYPTO -eq 1 || $SPDK_TEST_REDUCE -eq 1 ]]; then
-	# Make sure that memory is distributed across all NUMA nodes - by default, all goes to
-	# node0, but if QAT devices are attached to a different node, all of their VFs will end
-	# up under that node too and memory needs to be available there for the tests.
-	CLEAR_HUGE=yes HUGE_EVEN_ALLOC=yes ./scripts/setup.sh
-	./scripts/setup.sh status
-	if [[ $SPDK_TEST_USE_IGB_UIO -eq 1 ]]; then
+	if grep -q '#define SPDK_CONFIG_IGB_UIO_DRIVER 1' $rootdir/include/spdk/config.h; then
 		./scripts/qat_setup.sh igb_uio
 	else
 		./scripts/qat_setup.sh
@@ -187,7 +179,6 @@ if [ $SPDK_RUN_FUNCTIONAL_TEST -eq 1 ]; then
 	run_test "spdkcli_tcp" test/spdkcli/tcp.sh
 	run_test "dpdk_mem_utility" test/dpdk_memory_utility/test_dpdk_mem_info.sh
 	run_test "event" test/event/event.sh
-	run_test "accel_engine" test/accel_engine/accel_engine.sh
 
 	if [ $SPDK_TEST_BLOCKDEV -eq 1 ]; then
 		run_test "blockdev_general" test/bdev/blockdev.sh
@@ -195,7 +186,6 @@ if [ $SPDK_RUN_FUNCTIONAL_TEST -eq 1 ]; then
 		run_test "bdevperf_config" test/bdev/bdevperf/test_config.sh
 		if [[ $(uname -s) == Linux ]]; then
 			run_test "spdk_dd" test/dd/dd.sh
-			run_test "reactor_set_interrupt" test/interrupt/reactor_set_interrupt.sh
 		fi
 	fi
 
@@ -207,8 +197,8 @@ if [ $SPDK_RUN_FUNCTIONAL_TEST -eq 1 ]; then
 		run_test "blockdev_nvme" test/bdev/blockdev.sh "nvme"
 		run_test "blockdev_nvme_gpt" test/bdev/blockdev.sh "gpt"
 		run_test "nvme" test/nvme/nvme.sh
-		if [[ $SPDK_TEST_NVME_PMR -eq 1 ]]; then
-			run_test "nvme_pmr" test/nvme/nvme_pmr.sh
+		if [[ $SPDK_TEST_NVME_CLI -eq 1 ]]; then
+			run_test "nvme_cli" test/nvme/spdk_nvme_cli.sh
 		fi
 		if [[ $SPDK_TEST_NVME_CUSE -eq 1 ]]; then
 			run_test "nvme_cuse" test/nvme/cuse/nvme_cuse.sh
@@ -248,19 +238,12 @@ if [ $SPDK_RUN_FUNCTIONAL_TEST -eq 1 ]; then
 		# The NVMe-oF run test cases are split out like this so that the parser that compiles the
 		# list of all tests can properly differentiate them. Please do not merge them into one line.
 		if [ "$SPDK_TEST_NVMF_TRANSPORT" = "rdma" ]; then
-			timing_enter rdma_setup
-			rdma_device_init
-			timing_exit rdma_setup
 			run_test "nvmf_rdma" ./test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
-			run_test "spdkcli_nvmf_rdma" ./test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
+			run_test "spdkcli_nvmf_rdma" ./test/spdkcli/nvmf.sh
 		elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "tcp" ]; then
-			timing_enter tcp_setup
-			tcp_device_init
-			timing_exit tcp_setup
 			run_test "nvmf_tcp" ./test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
-			run_test "spdkcli_nvmf_tcp" ./test/spdkcli/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
+			run_test "spdkcli_nvmf_tcp" ./test/spdkcli/nvmf.sh
 			run_test "nvmf_identify_passthru" test/nvmf/target/identify_passthru.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
-			run_test "nvmf_dif" test/nvmf/target/dif.sh
 		elif [ "$SPDK_TEST_NVMF_TRANSPORT" = "fc" ]; then
 			run_test "nvmf_fc" ./test/nvmf/nvmf.sh --transport=$SPDK_TEST_NVMF_TRANSPORT
 			run_test "spdkcli_nvmf_fc" ./test/spdkcli/nvmf.sh
@@ -327,10 +310,6 @@ if [ $SPDK_RUN_FUNCTIONAL_TEST -eq 1 ]; then
 			run_test "blockdev_crypto_qat" ./test/bdev/blockdev.sh "crypto_qat"
 		fi
 	fi
-
-	if [[ $SPDK_TEST_SCHEDULER -eq 1 ]]; then
-		run_test "scheduler" ./test/scheduler/scheduler.sh
-	fi
 fi
 
 timing_enter cleanup
@@ -345,7 +324,7 @@ trap - SIGINT SIGTERM EXIT
 # catch any stray core files
 process_core
 
-if hash lcov && ! [[ "$CC_TYPE" == *"clang"* ]]; then
+if hash lcov; then
 	# generate coverage data and combine with baseline
 	$LCOV -q -c -d $src -t "$(hostname)" -o $out/cov_test.info
 	$LCOV -q -a $out/cov_base.info -a $out/cov_test.info -o $out/cov_total.info

configure

@@ -7,7 +7,8 @@ trap 'echo -e "\n\nConfiguration failed\n\n" >&2' ERR
 rootdir=$(readlink -f $(dirname $0))
 source "$rootdir/scripts/common.sh"
 
-function usage() {
+function usage()
+{
 	echo "'configure' configures SPDK to compile on supported platforms."
 	echo ""
 	echo "Usage: ./configure [OPTION]..."
@@ -31,7 +32,6 @@ function usage() {
 	echo " --enable-lto              Enable link-time optimization"
 	echo " --enable-pgo-capture      Enable generation of profile guided optimization data"
 	echo " --enable-pgo-use          Use previously captured profile guided optimization data"
-	echo " --enable-cet              Enable Intel Control-flow Enforcement Technology (CET)"
 	echo " --disable-tests           Disable building of functional tests"
 	echo " --disable-unit-tests      Disable building of unit tests"
 	echo " --disable-examples        Disable building of examples"
@@ -48,6 +48,10 @@ function usage() {
 	echo "                           example: /usr/share/dpdk/x86_64-default-linuxapp-gcc"
 	echo " env                       Use an alternate environment implementation instead of DPDK."
 	echo "                           Implies --without-dpdk."
+	echo " igb-uio-driver            Build DPDK's igb-uio driver."
+	echo "                           Required on some systems to use qat devices. This flag is"
+	echo "                           effective only with the default dpdk submodule."
+	echo "                           No path required"
 	echo " idxd                      Build the IDXD library and accel framework plug-in module."
 	echo "                           Disabled while experimental. Only built for x86 when enabled."
 	echo " crypto                    Build vbdev crypto module."
@@ -58,8 +62,6 @@ function usage() {
 	echo "                           No path required."
 	echo " virtio                    Build vhost initiator and virtio-pci bdev modules."
 	echo "                           No path required."
-	echo " vfio-user                 Build custom vfio-user transport for NVMf target and NVMe initiator."
-	echo "                           example: /usr/src/libvfio-user"
 	echo " pmdk                      Build persistent memory bdev."
 	echo "                           example: /usr/share/pmdk"
 	echo " reduce                    Build vbdev compression module."
@@ -96,8 +98,6 @@ function usage() {
 	echo "                           No path required."
 	echo " raid5                     Build with bdev_raid module RAID5 support."
 	echo "                           No path required."
-	echo " wpdk                      Build using WPDK to provide support for Windows (experimental)."
-	echo "                           The argument must be a directory containing lib and include."
 	echo ""
 	echo "Environment variables:"
 	echo ""
@@ -133,18 +133,14 @@ for i in "$@"; do
 done
 
 # Detect the compiler toolchain
-$rootdir/scripts/detect_cc.sh --cc="$CC" --cxx="$CXX" --lto="${CONFIG[LTO]}" --ld="$LD" --cross-prefix="${CONFIG[CROSS_PREFIX]}" > $rootdir/mk/cc.mk
+$rootdir/scripts/detect_cc.sh --cc="$CC" --cxx="$CXX" --lto="${CONFIG[LTO]}" --ld="$LD" --cross-prefix="${CONFIG[CROSS_PREFIX]}"  > $rootdir/mk/cc.mk
 
-CC=$(grep "DEFAULT_CC=" "$rootdir/mk/cc.mk" | sed s/DEFAULT_CC=//)
-CC_TYPE=$(grep "CC_TYPE=" "$rootdir/mk/cc.mk" | cut -d "=" -f 2)
+CC=$(cat $rootdir/mk/cc.mk | grep "DEFAULT_CC=" | sed s/DEFAULT_CC=//)
+CC_TYPE=$(cat $rootdir/mk/cc.mk | grep "CC_TYPE=" | cut -d "=" -f 2)
 
 arch=$($CC -dumpmachine)
 sys_name=$(uname -s)
 
-if [[ $arch == *mingw* ]] || [[ $arch == *windows* ]]; then
-	sys_name=Windows
-fi
-
 # Sanitize default configuration. All parameters set by user explicit should fail
 # Force no ISA-L if non-x86 or non-aarch64 architecture
 if [[ "${CONFIG[ISAL]}" = "y" ]]; then
@@ -154,7 +150,7 @@ if [[ "${CONFIG[ISAL]}" = "y" ]]; then
 	fi
 fi
 
-if [[ $sys_name != "Linux" ]]; then
+if [[ $sys_name == "FreeBSD" ]]; then
 	# Vhost, rte_vhost library and virtio are only supported on Linux.
 	CONFIG[VHOST]="n"
 	CONFIG[VIRTIO]="n"
@@ -163,7 +159,7 @@ fi
 
 #check nasm only on x86
 if [[ $arch == x86_64* ]]; then
-	ver=$(nasm -v 2> /dev/null | awk '{print $3}')
+	ver=$(nasm -v 2>/dev/null | awk '{print $3}')
 	if lt "$ver" 2.14; then
 		# ISA-L, compression & crypto require NASM version 2.14 or newer.
 		CONFIG[ISAL]=n
@@ -188,7 +184,7 @@ function check_dir() {
 
 for i in "$@"; do
 	case "$i" in
-		-h | --help)
+		-h|--help)
 			usage
 			exit 0
 			;;
@@ -269,12 +265,6 @@ for i in "$@"; do
 		--disable-werror)
 			CONFIG[WERROR]=n
 			;;
-		--enable-cet)
-			CONFIG[CET]=y
-			;;
-		--disable-cet)
-			CONFIG[CET]=n
-			;;
 		--with-dpdk=*)
 			check_dir "$i"
 			CONFIG[DPDK_DIR]=$(readlink -f ${i#*=})
@@ -282,10 +272,6 @@ for i in "$@"; do
 		--without-dpdk)
 			CONFIG[DPDK_DIR]=
 			;;
-		--with-wpdk=*)
-			check_dir "$i"
-			CONFIG[WPDK_DIR]=$(readlink -f ${i#*=})
-			;;
 		--with-env=*)
 			CONFIG[ENV]="${i#*=}"
 			;;
@@ -348,18 +334,6 @@ for i in "$@"; do
 		--without-virtio)
 			CONFIG[VIRTIO]=n
 			;;
-		--with-vfio-user)
-			CONFIG[VFIO_USER]=y
-			CONFIG[VFIO_USER_DIR]=""
-			;;
-		--with-vfio-user=*)
-			CONFIG[VFIO_USER]=y
-			check_dir "$i"
-			CONFIG[VFIO_USER_DIR]=$(readlink -f ${i#*=})
-			;;
-		--without-vfio-user)
-			CONFIG[VFIO_USER]=n
-			;;
 		--with-pmdk)
 			CONFIG[PMDK]=y
 			CONFIG[PMDK_DIR]=""
@@ -398,6 +372,12 @@ for i in "$@"; do
 			CONFIG[VTUNE_DIR]=
 			CONFIG[VTUNE]=n
 			;;
+		--with-igb-uio-driver)
+			CONFIG[IGB_UIO_DRIVER]=y
+			;;
+		--without-igb-uio-driver)
+			CONFIG[IGB_UIO_DRIVER]=n
+			;;
 		--with-ocf)
 			CONFIG[OCF]=y
 			CONFIG[OCF_PATH]=$(readlink -f "./ocf")
@@ -459,36 +439,16 @@ for i in "$@"; do
 			echo "Unrecognized option $i"
 			usage
 			exit 1
-			;;
 	esac
 done
 
 if [[ $arch == x86_64* ]]; then
-	BUILD_CMD=("$CC" -o /dev/null -x c $CPPFLAGS $CFLAGS $LDFLAGS "-march=native")
+	BUILD_CMD=($CC -o /dev/null -x c $CPPFLAGS $CFLAGS $LDFLAGS -march=native)
 else
-	BUILD_CMD=("$CC" -o /dev/null -x c $CPPFLAGS $CFLAGS $LDFLAGS)
+	BUILD_CMD=($CC -o /dev/null -x c $CPPFLAGS $CFLAGS $LDFLAGS)
 fi
 BUILD_CMD+=(-I/usr/local/include -L/usr/local/lib)
 
-if [[ "${CONFIG[VFIO_USER]}" = "y" ]]; then
-
-	if ! hash cmake; then
-		echo "ERROR: --with-vfio-user requires cmake"
-		echo "Please install then re-run this script"
-		exit 1
-	fi
-	if [[ ! -d /usr/include/json-c ]] && [[ ! -d /usr/local/include/json-c ]]; then
-		echo "ERROR: --with-vfio-user requires json-c-devel"
-		echo "Please install then re-run this script"
-		exit 1
-	fi
-	if [[ ! -e /usr/include/cmocka.h ]] && [[ ! -e /usr/local/include/cmocka.h ]]; then
-		echo "ERROR: --with-vfio-user requires libcmocka-devel"
-		echo "Please install then re-run this script"
-		exit 1
-	fi
-fi
-
 # IDXD uses Intel specific instructions.
 if [[ "${CONFIG[IDXD]}" = "y" ]]; then
 	if [ $(uname -s) == "FreeBSD" ]; then
@@ -496,7 +456,7 @@ if [[ "${CONFIG[IDXD]}" = "y" ]]; then
 		cpu_vendor=$(sysctl -a | grep hw.model | cut -c 1-15)
 	else
 		intel="GenuineIntel"
-		cpu_vendor=$(grep -i 'vendor' /proc/cpuinfo --max-count=1)
+		cpu_vendor=$(grep -i 'vendor' /proc/cpuinfo  --max-count=1)
 	fi
 	if [[ "$cpu_vendor" != *"$intel"* ]]; then
 		echo "ERROR: IDXD cannot be used due to CPU incompatiblity."
@@ -513,15 +473,15 @@ if [[ "${CONFIG[ISAL]}" = "y" ]]; then
 fi
 
 if [[ "${CONFIG[ISAL]}" = "n" ]] && [[ "${CONFIG[REDUCE]}" = "y" ]]; then
-	echo "ERROR Conflicting options: --with-reduce is not compatible with --without-isal."
-	exit 1
+		echo "ERROR Conflicting options: --with-reduce is not compatible with --without-isal."
+		exit 1
 fi
 
 if [ -z "${CONFIG[ENV]}" ]; then
 	CONFIG[ENV]=$rootdir/lib/env_dpdk
 	echo "Using default SPDK env in ${CONFIG[ENV]}"
 	if [ -z "${CONFIG[DPDK_DIR]}" ]; then
-		if [ ! -f "$rootdir"/dpdk/config/meson.build ]; then
+		if [ ! -f "$rootdir"/dpdk/config/common_base ]; then
 			echo "DPDK not found; please specify --with-dpdk=<path> or run:"
 			echo
 			echo "  git submodule update --init"
@@ -530,6 +490,29 @@ if [ -z "${CONFIG[ENV]}" ]; then
 			CONFIG[DPDK_DIR]="${rootdir}/dpdk/build"
 			echo "Using default DPDK in ${CONFIG[DPDK_DIR]}"
 		fi
+
+		if [ "${CONFIG[VHOST]}" = "y" ]; then
+			# We lookup "common_linux" file to check if DPDK version is >= 19.05.
+			# "common_linux" is available since exactly DPDK 19.05 - it was renamed
+			# from "common_linuxapp".
+			if [ ! -f "$rootdir"/dpdk/config/common_linux ]; then
+				echo "ERROR: Vhost only supports DPDK >= 19.05. Please use newer DPDK or disable vhost library --without-vhost."
+				exit 1
+			fi
+		fi
+	else
+		if [ "${CONFIG[VHOST]}" = "y" ]; then
+			# DPDK must be already built, so we can simply try to use the new rte_vhost.
+			# It has a number of internal dependencies though, so don't try to link the
+			# program, just compile it
+			if ! echo -e '#include <rte_vhost.h>\n' \
+				'int main(void) { return rte_vhost_extern_callback_register(0, NULL, NULL); }\n' \
+				| ${BUILD_CMD[@]} -c -Wno-deprecated-declarations -Werror \
+				-I"${CONFIG[DPDK_DIR]}/include" - &>/dev/null; then
+				echo "ERROR: Vhost only supports DPDK >= 19.05. Please use newer DPDK or disable vhost library --without-vhost."
+				exit 1
+			fi
+		fi
 	fi
 else
 	if [ -n "${CONFIG[DPDK_DIR]}" ]; then
@@ -549,23 +532,6 @@ else
 	CONFIG[VIRTIO]="n"
 fi
 
-if [[ $sys_name == "Windows" ]]; then
-	if [ -z "${CONFIG[WPDK_DIR]}" ]; then
-		if [ ! -f "$rootdir"/wpdk/Makefile ]; then
-			echo "WPDK not found; please specify --with-wpdk=<path>. See https://wpdk.github.io."
-			exit 1
-		else
-			CONFIG[WPDK_DIR]="${rootdir}/wpdk/build"
-			echo "Using default WPDK in ${CONFIG[WPDK_DIR]}"
-		fi
-	fi
-else
-	if [ -n "${CONFIG[WPDK_DIR]}" ]; then
-		echo "ERROR: --with-wpdk is only supported for Windows"
-		exit 1
-	fi
-fi
-
 if [ "${CONFIG[VTUNE]}" = "y" ]; then
 	if [ -z "${CONFIG[VTUNE_DIR]}" ]; then
 		echo "When VTune is enabled, you must specify the VTune directory using --with-vtune=path"
@@ -573,7 +539,7 @@ if [ "${CONFIG[VTUNE]}" = "y" ]; then
 	fi
 fi
 
-if [[ "${CONFIG[ASAN]}" = "y" && "${CONFIG[TSAN]}" = "y" ]]; then
+if [ "${CONFIG[ASAN]}" = "y" -a "${CONFIG[TSAN]}" = "y" ]; then
 	echo "ERROR: ASAN and TSAN cannot be enabled at the same time."
 	exit 1
 fi
@@ -586,7 +552,7 @@ if [[ $sys_name == "FreeBSD" ]]; then
 	fi
 fi
 
-if [[ $sys_name != "Linux" ]]; then
+if [[ $sys_name == "FreeBSD" ]]; then
 	if [[ "${CONFIG[VHOST]}" == "y" ]]; then
 		echo "Vhost is only supported on Linux."
 		exit 1
@@ -598,22 +564,22 @@ if [[ $sys_name != "Linux" ]]; then
 fi
 
 if [ "${CONFIG[RDMA]}" = "y" ]; then
-	if [[ ! "${CONFIG[RDMA_PROV]}" == "verbs" ]] && [[ ! "${CONFIG[RDMA_PROV]}" == "mlx5_dv" ]]; then
-		echo "Invalid RDMA provider specified, must be \"verbs\" or \"mlx5_dv\""
-		exit 1
-	fi
+  if [[ ! "${CONFIG[RDMA_PROV]}" == "verbs" ]] && [[ ! "${CONFIG[RDMA_PROV]}" == "mlx5_dv" ]]; then
+    echo "Invalid RDMA provider specified, must be \"verbs\" or \"mlx5_dv\""
+    exit 1
+  fi
 
 	if ! echo -e '#include <infiniband/verbs.h>\n#include <rdma/rdma_verbs.h>\n' \
 		'int main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -libverbs -lrdmacm - 2> /dev/null; then
-		echo "--with-rdma requires libverbs and librdmacm."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -libverbs -lrdmacm - 2>/dev/null; then
+		echo --with-rdma requires libverbs and librdmacm.
+		echo Please install then re-run this script.
 		exit 1
 	fi
 
 	if echo -e '#include <infiniband/verbs.h>\n' \
 		'int main(void) { return !!IBV_WR_SEND_WITH_INV; }\n' \
-		| "${BUILD_CMD[@]}" -c - 2> /dev/null; then
+		| ${BUILD_CMD[@]} -c - 2>/dev/null; then
 		CONFIG[RDMA_SEND_WITH_INVAL]="y"
 	else
 		CONFIG[RDMA_SEND_WITH_INVAL]="n"
@@ -635,7 +601,7 @@ than or equal to 4.14 will see significantly reduced performance.
 
 	if echo -e '#include <rdma/rdma_cma.h>\n' \
 		'int main(void) { return !!RDMA_OPTION_ID_ACK_TIMEOUT; }\n' \
-		| "${BUILD_CMD[@]}" -c - 2> /dev/null; then
+		| ${BUILD_CMD[@]} -c - 2>/dev/null; then
 		CONFIG[RDMA_SET_ACK_TIMEOUT]="y"
 	else
 		CONFIG[RDMA_SET_ACK_TIMEOUT]="n"
@@ -643,18 +609,18 @@ than or equal to 4.14 will see significantly reduced performance.
 	fi
 
 	if [ "${CONFIG[RDMA_PROV]}" == "mlx5_dv" ]; then
-		if ! echo -e '#include <spdk/stdinc.h>\n' \
-			'#include <infiniband/mlx5dv.h>\n' \
-			'#include <rdma/rdma_cma.h>\n' \
-			'int main(void) { return rdma_establish(NULL) || ' \
-			'!!IBV_QP_INIT_ATTR_SEND_OPS_FLAGS || !!MLX5_OPCODE_RDMA_WRITE; }\n' \
-			| "${BUILD_CMD[@]}" -lmlx5 -I${rootdir}/include -c - 2> /dev/null; then
-			echo "mlx5_dv provider is not supported"
-			exit 1
-		fi
+	  if ! echo -e '#include <spdk/stdinc.h>\n' \
+	    '#include <infiniband/mlx5dv.h>\n' \
+	    '#include <rdma/rdma_cma.h>\n' \
+		  'int main(void) { return rdma_establish(NULL) || ' \
+		  '!!IBV_QP_INIT_ATTR_SEND_OPS_FLAGS || !!MLX5_OPCODE_RDMA_WRITE; }\n' \
+		  | ${BUILD_CMD[@]} -lmlx5 -I${rootdir}/include -c - 2>/dev/null; then
+      echo "mlx5_dv provider is not supported"
+		  exit 1
+	  fi
 	fi
 
-	echo "Using '${CONFIG[RDMA_PROV]}' RDMA provider"
+	echo "Using "${CONFIG[RDMA_PROV]}" RDMA provider"
 fi
 
 if [[ "${CONFIG[FC]}" = "y" ]]; then
@@ -678,29 +644,45 @@ if [[ "${CONFIG[ISAL]}" = "y" ]] || [[ "${CONFIG[CRYPTO]}" = "y" ]]; then
 	fi
 fi
 
+if [[ "${CONFIG[ISAL]}" = "y" ]]; then
+	if [ ! -f "$rootdir"/isa-l/autogen.sh ]; then
+		echo "ISA-L was not found; To install ISA-L run:"
+		echo "  git submodule update --init"
+		exit 1
+	fi
+
+	cd $rootdir/isa-l
+	ISAL_LOG=$rootdir/isa-l/spdk-isal.log
+	echo -n "Configuring ISA-L (logfile: $ISAL_LOG)..."
+	./autogen.sh &> $ISAL_LOG
+	./configure CFLAGS="-fPIC -g -O2" --enable-shared=no >> $ISAL_LOG 2>&1
+	echo "done."
+	cd $rootdir
+fi
+
 if [[ "${CONFIG[PMDK]}" = "y" ]]; then
 	if ! echo -e '#include <libpmemblk.h>\nint main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -lpmemblk - 2> /dev/null; then
-		echo "--with-pmdk requires libpmemblk."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -lpmemblk - 2>/dev/null; then
+		echo --with-pmdk requires libpmemblk.
+		echo Please install then re-run this script.
 		exit 1
 	fi
 fi
 
 if [[ "${CONFIG[REDUCE]}" = "y" ]]; then
 	if ! echo -e '#include <libpmem.h>\nint main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -lpmem - 2> /dev/null; then
-		echo "--with-reduce requires libpmem."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -lpmem - 2>/dev/null; then
+		echo --with-reduce requires libpmem.
+		echo Please install then re-run this script.
 		exit 1
 	fi
 fi
 
 if [[ "${CONFIG[NVME_CUSE]}" = "y" ]]; then
 	if ! echo -e '#define FUSE_USE_VERSION 31\n#include <fuse3/cuse_lowlevel.h>\n#include <fuse3/fuse_lowlevel.h>\n#include <fuse3/fuse_opt.h>\nint main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -lfuse3 -D_FILE_OFFSET_BITS=64 - 2> /dev/null; then
-		echo "--with-cuse requires libfuse3."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -lfuse3 -D_FILE_OFFSET_BITS=64 - 2>/dev/null; then
+		echo --with-cuse requires libfuse3.
+		echo Please install then re-run this script.
 		exit 1
 	fi
 fi
@@ -708,9 +690,9 @@ fi
 if [[ "${CONFIG[RBD]}" = "y" ]]; then
 	if ! echo -e '#include <rbd/librbd.h>\n#include <rados/librados.h>\n' \
 		'int main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -lrados -lrbd - 2> /dev/null; then
-		echo "--with-rbd requires librados and librbd."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -lrados -lrbd - 2>/dev/null; then
+		echo --with-rbd requires librados and librbd.
+		echo Please install then re-run this script.
 		exit 1
 	fi
 fi
@@ -722,39 +704,39 @@ if [[ "${CONFIG[ISCSI_INITIATOR]}" = "y" ]]; then
 		'#error\n' \
 		'#endif\n' \
 		'int main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -L/usr/lib64/iscsi -liscsi - 2> /dev/null; then
-		echo "--with-iscsi-initiator requires libiscsi with"
-		echo "LIBISCSI_API_VERSION >= 20150621."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -L/usr/lib64/iscsi -liscsi - 2>/dev/null; then
+		echo --with-iscsi-initiator requires libiscsi with
+		echo 'LIBISCSI_API_VERSION >= 20150621.'
+		echo Please install then re-run this script.
 		exit 1
 	fi
 fi
 
 if [[ "${CONFIG[ASAN]}" = "y" ]]; then
 	if ! echo -e 'int main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -fsanitize=address - 2> /dev/null; then
-		echo "--enable-asan requires libasan."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -fsanitize=address - 2>/dev/null; then
+		echo --enable-asan requires libasan.
+		echo Please install then re-run this script.
 		exit 1
 	fi
 fi
 
 if [[ "${CONFIG[UBSAN]}" = "y" ]]; then
 	if ! echo -e 'int main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -fsanitize=undefined - 2> /dev/null; then
-		echo "--enable-ubsan requires libubsan."
-		echo "Please install then re-run this script."
-		echo "If installed, please check that the GCC version is at least 6.4"
-		echo "and synchronize CC accordingly."
+		| ${BUILD_CMD[@]} -fsanitize=undefined - 2>/dev/null; then
+		echo --enable-ubsan requires libubsan.
+		echo Please install then re-run this script.
+		echo If installed, please check that the GCC version is at least 6.4 \
+		     and synchronize CC accordingly.
 		exit 1
 	fi
 fi
 
 if [[ "${CONFIG[TSAN]}" = "y" ]]; then
 	if ! echo -e 'int main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -fsanitize=thread - 2> /dev/null; then
-		echo "--enable-tsan requires libtsan."
-		echo "Please install then re-run this script."
+		| ${BUILD_CMD[@]} -fsanitize=thread - 2>/dev/null; then
+		echo --enable-tsan requires libtsan.
+		echo Please install then re-run this script.
 		exit 1
 	fi
 fi
@@ -785,11 +767,6 @@ if [[ "${CONFIG[URING]}" = "y" ]]; then
 			echo "${CONFIG[URING_PATH]}: directory not found"
 			exit 1
 		fi
-	elif ! echo -e '#include <liburing.h>\nint main(void) { return 0; }\n' \
-		| "${BUILD_CMD[@]}" -luring - 2> /dev/null; then
-		echo "--with-uring requires liburing."
-		echo "Please build and install then re-run this script."
-		exit 1
 	fi
 fi
 
@@ -801,51 +778,22 @@ if [[ "${CONFIG[FUSE]}" = "y" ]]; then
 	fi
 fi
 
-if [ "${CONFIG[CET]}" = "y" ]; then
-	if ! echo -e 'int main(void) { return 0; }\n' | "${BUILD_CMD[@]}" -fcf-protection - 2> /dev/null; then
-		echo "--enable-cet requires compiler/linker that supports CET."
-		echo "Please install then re-run this script."
-		exit 1
-	fi
-fi
-
-if [[ "${CONFIG[ISAL]}" = "y" ]]; then
-	if [ ! -f "$rootdir"/isa-l/autogen.sh ]; then
-		echo "ISA-L was not found; To install ISA-L run:"
-		echo "  git submodule update --init"
-		exit 1
-	fi
-
-	cd $rootdir/isa-l
-	ISAL_LOG=$rootdir/isa-l/spdk-isal.log
-	if [[ -n "${CONFIG[CROSS_PREFIX]}" ]]; then
-		ISAL_OPTS=("--host=${CONFIG[CROSS_PREFIX]}")
-	else
-		ISAL_OPTS=()
-	fi
-	echo -n "Configuring ISA-L (logfile: $ISAL_LOG)..."
-	./autogen.sh &> $ISAL_LOG
-	./configure CFLAGS="-fPIC -g -O2" "${ISAL_OPTS[@]}" --enable-shared=no >> $ISAL_LOG 2>&1
-	echo "done."
-	cd $rootdir
-fi
-
 # We are now ready to generate final configuration. But first do sanity
 # check to see if all keys in CONFIG array have its reflection in CONFIG file.
-if (($(grep -cE "^\s*CONFIG_[[:alnum:]_]+=" "$rootdir/CONFIG") != ${#CONFIG[@]})); then
+if [ $(egrep -c "^\s*CONFIG_[[:alnum:]_]+=" $rootdir/CONFIG) -ne ${#CONFIG[@]} ]; then
 	echo ""
 	echo "BUG: Some configuration options are not present in CONFIG file. Please update this file."
 	echo "Missing options in CONFIG (+) file and in current config (-): "
 	diff -u --label "CONFIG file" --label "CONFIG[@]" \
 		<(sed -r -e '/^\s*$/d; /^\s*#.*/d; s/(CONFIG_[[:alnum:]_]+)=.*/\1/g' CONFIG | sort) \
-		<(printf "CONFIG_%s\n" "${!CONFIG[@]}" | sort)
+		<(printf "CONFIG_%s\n" ${!CONFIG[@]} | sort)
 	exit 1
 fi
 
 echo -n "Creating mk/config.mk..."
 cp -f $rootdir/CONFIG $rootdir/mk/config.mk
-for key in "${!CONFIG[@]}"; do
-	sed -i.bak -r "s#[[:space:]]*CONFIG_${key}=.*#CONFIG_${key}\?=${CONFIG[$key]}#g" $rootdir/mk/config.mk
+for key in ${!CONFIG[@]}; do
+	sed -i.bak -r "s#^\s*CONFIG_${key}=.*#CONFIG_${key}\?=${CONFIG[$key]}#g" $rootdir/mk/config.mk
 done
 # On FreeBSD sed -i 'SUFFIX' - SUFFIX is mandatory. So no way but to delete the backed file.
 rm -f $rootdir/mk/config.mk.bak
@@ -863,7 +811,7 @@ echo "done."
 # Create .sh with build config for easy sourcing|lookup during the tests.
 for conf in "${!CONFIG[@]}"; do
 	echo "CONFIG_$conf=${CONFIG[$conf]}"
-done > "$rootdir/test/common/build_config.sh"
+done >"$rootdir/test/common/build_config.sh"
 
 if [[ $sys_name == "FreeBSD" ]]; then
 	echo "Type 'gmake' to build."

deprecation.md

@@ -1,42 +0,0 @@
-# ABI and API Deprecation {#deprecation}
-
-This document details the policy for maintaining stability of SPDK ABI and API.
-
-Major ABI version can change at most once for each quarterly SPDK release.
-ABI versions are managed separately for each library and follow [Semantic Versoning](https://semver.org/).
-
-API and ABI deprecation notices shall be posted in the next section.
-Each entry must describe what will be removed and can suggest the future use or alternative.
-Specific future SPDK release for the removal must be provided.
-ABI cannot be removed without providing deprecation notice for at least single SPDK release.
-
-# Deprecation Notices {#deprecation-notices}
-
-## net
-
-The net library is deprecated and will be removed in the 21.07 release.
-
-## nvmf
-
-The following APIs have been deprecated and will be removed in SPDK 21.07:
-- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`),
-- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`),
-- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`),
-- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`),
-- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`),
-- `poll_group_get_stat` (transport op in `nvmf_transport.h`),
-- `poll_group_free_stat` (transport op in `nvmf_transport.h`).
-Please use `spdk_nvmf_poll_group_dump_stat` and `poll_group_dump_stat` instead.
-
-## rpc
-
-Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07,
-use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead.
-Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07,
-use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead.
-
-## rpm
-
-`pkg/spdk.spec` is considered to be deprecated and scheduled for removal in SPDK 21.07.
-Please use `rpmbuild/spdk.spec` instead and see
-[RPM documentation](https://spdk.io/doc/rpm.html) for more details.

doc/Doxyfile

@@ -234,7 +234,7 @@ ALIASES                =
 # A mapping has the form "name=value". For example adding "class=itcl::class"
 # will allow you to use the command class in the itcl::class meaning.
 
-# TCL_SUBST              =
+TCL_SUBST              =
 
 # Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources
 # only. Doxygen will then generate output that is more tailored for C. For
@@ -795,7 +795,6 @@ INPUT                  += \
                          misc.md \
                          driver_modules.md \
                          tools.md \
-                         ci_tools.md \
                          performance_reports.md \
 
 # All remaining pages are listed here in alphabetical order by filename.
@@ -813,7 +812,6 @@ INPUT                  += \
                          compression.md \
                          concurrency.md \
                          containers.md \
-                         ../deprecation.md \
                          event.md \
                          ftl.md \
                          gdb_macros.md \
@@ -828,17 +826,14 @@ INPUT                  += \
                          memory.md \
                          notify.md \
                          nvme.md \
+                         nvme-cli.md \
                          nvme_spec.md \
                          nvmf.md \
                          nvmf_tgt_pg.md \
                          nvmf_tracing.md \
                          overview.md \
                          peer_2_peer.md \
-                         pkgconfig.md \
                          porting.md \
-                         rpm.md \
-                         scheduler.md \
-                         shfmt.md \
                          spdkcli.md \
                          spdk_top.md \
                          ssd_internals.md \
@@ -1105,7 +1100,7 @@ ALPHABETICAL_INDEX     = YES
 # Minimum value: 1, maximum value: 20, default value: 5.
 # This tag requires that the tag ALPHABETICAL_INDEX is set to YES.
 
-# COLS_IN_ALPHA_INDEX    = 5
+COLS_IN_ALPHA_INDEX    = 5
 
 # In case all classes in a project start with a common prefix, all classes will
 # be put under the same header in the alphabetical index. The IGNORE_PREFIX tag
@@ -1666,7 +1661,7 @@ EXTRA_SEARCH_MAPPINGS  =
 # If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output.
 # The default value is: YES.
 
-GENERATE_LATEX         = NO
+GENERATE_LATEX         = YES
 
 # The LATEX_OUTPUT tag is used to specify where the LaTeX docs will be put. If a
 # relative path is entered the value of OUTPUT_DIRECTORY will be put in front of
@@ -2170,7 +2165,7 @@ EXTERNAL_PAGES         = YES
 # interpreter (i.e. the result of 'which perl').
 # The default file (with absolute path) is: /usr/bin/perl.
 
-# PERL_PATH              = /usr/bin/perl
+PERL_PATH              = /usr/bin/perl
 
 #---------------------------------------------------------------------------
 # Configuration options related to the dot tool
@@ -2192,7 +2187,7 @@ CLASS_DIAGRAMS         = YES
 # the mscgen tool resides. If left empty the tool is assumed to be found in the
 # default search path.
 
-# MSCGEN_PATH            =
+MSCGEN_PATH            =
 
 # You can include diagrams made with dia in doxygen documentation. Doxygen will
 # then run dia to produce the diagram and insert it in the documentation. The

doc/accel_fw.md

@@ -66,14 +66,14 @@ To use the IOAT engine, use the RPC [`ioat_scan_accel_engine`](https://spdk.io/d
 To use the DSA engine, use the RPC [`idxd_scan_accel_engine`](https://spdk.io/doc/jsonrpc.html) with an optional parameter of `-c` and provide a configuration number of either 0 or 1. These pre-defined configurations determine how the DSA engine will be setup in terms
 of work queues and engines.  The DSA engine is very flexible allowing for various configurations of these elements to either account for different quality of service requirements or to isolate hardware paths where the back end media is of varying latency (i.e. persistent memory vs DRAM).  The pre-defined configurations are as follows:
 
-0: A single work queue backed with four DSA engines.  This is a generic configuration
-that enables the hardware to best determine which engine to use as it pulls in new
-operations.
+0: Four separate work queues each backed with one DSA engine.  This is a generic
+configuration that provides 4 portals to submit operations to each with a
+single engine behind it providing some level of isolation as operations are
+submitted round-robin.
 
-1: Two separate work queues each backed with two DSA engines. This is another
-generic configuration that is documented in the specification and allows the
-application to partition submissions across two work queues. This would be useful
-when different priorities might be desired per group.
+1: Two separate work queues each backed with two DSA engines.  This is another
+generic configuration that provides 2 portals to submit operations to and
+lets the DSA hardware decide which engine to select based on loading.
 
 There are several other configurations that are possible that include quality
 of service parameters on the work queues that are not currently utilized by

doc/applications.md

@@ -35,14 +35,14 @@ Param    | Long Param             | Type     | Default                | Descript
 -i       | --shm-id               | integer  |                        | @ref cmd_arg_multi_process
 -m       | --cpumask              | CPU mask | 0x1                    | application @ref cpu_mask
 -n       | --mem-channels         | integer  | all channels           | number of memory channels used for DPDK
--p       | --main-core            | integer  | first core in CPU mask | main (primary) core for DPDK
+-p       | --master-core          | integer  | first core in CPU mask | master (primary) core for DPDK
 -r       | --rpc-socket           | string   | /var/tmp/spdk.sock     | RPC listen address
 -s       | --mem-size             | integer  | all hugepage memory    | @ref cmd_arg_memory_size
 |        | --silence-noticelog    | flag     |                        | disable notice level logging to `stderr`
 -u       | --no-pci               | flag     |                        | @ref cmd_arg_disable_pci_access.
 |        | --wait-for-rpc         | flag     |                        | @ref cmd_arg_deferred_initialization
--B       | --pci-blocked          | B:D:F    |                        | @ref cmd_arg_pci_blocked_allowed.
--A       | --pci-allowed          | B:D:F    |                        | @ref cmd_arg_pci_blocked_allowed.
+-B       | --pci-blacklist        | B:D:F    |                        | @ref cmd_arg_pci_blacklist_whitelist.
+-W       | --pci-whitelist        | B:D:F    |                        | @ref cmd_arg_pci_blacklist_whitelist.
 -R       | --huge-unlink          | flag     |                        | @ref cmd_arg_huge_unlink
 |        | --huge-dir             | string   | the first discovered   | allocate hugepages from a specific mount
 -L       | --logflag              | string   |                        | @ref cmd_arg_log_flags
@@ -121,12 +121,12 @@ If SPDK is run with PCI access disabled it won't detect any PCI devices. This
 includes primarily NVMe and IOAT devices. Also, the VFIO and UIO kernel modules
 are not required in this mode.
 
-### PCI address blocked and allowed lists {#cmd_arg_pci_blocked_allowed}
+### PCI address blacklist and whitelist {#cmd_arg_pci_blacklist_whitelist}
 
-If blocked list is used, then all devices with the provided PCI address will be
-ignored. If an allowed list is used, only allowed devices will be probed.
-`-B` or `-A` can be used more than once, but cannot be mixed together. That is,
-`-B` and `-A` cannot be used at the same time.
+If blacklist is used, then all devices with the provided PCI address will be
+ignored. If a whitelist is used, only whitelisted devices will be probed.
+`-B` or `-W` can be used more than once, but cannot be mixed together. That is,
+`-B` and `-W` cannot be used at the same time.
 
 ### Unlink hugepage files after initialization {#cmd_arg_huge_unlink}
 

doc/bdev.md

@@ -312,8 +312,10 @@ To remove `Cache1`:
 
 During removal OCF-cache will be stopped and all cached data will be written to the core device.
 
-Note that OCF has a per-device RAM requirement. More details can be found in the
-[OCF documentation](https://open-cas.github.io/guide_system_requirements.html).
+Note that OCF has a per-device RAM requirement
+of about 56000 + _cache device size_ * 58 / _cache line size_ (in bytes).
+To get more information on OCF
+please visit [OCF documentation](https://open-cas.github.io/).
 
 # Malloc bdev {#bdev_config_malloc}
 
@@ -369,22 +371,15 @@ This command will remove NVMe bdev named Nvme0.
 
 ## NVMe bdev character device {#bdev_config_nvme_cuse}
 
-This feature is considered as experimental. You must configure with --with-nvme-cuse
-option to enable this RPC.
+This feature is considered as experimental.
 
 Example commands
 
-`rpc.py bdev_nvme_cuse_register -n Nvme3
+`rpc.py bdev_nvme_cuse_register -n Nvme0 -p spdk/nvme0`
 
-This command will register a character device under /dev/spdk associated with Nvme3
-controller. If there are namespaces created on Nvme3 controller, a namespace
-character device is also created for each namespace.
-
-For example, the first controller registered will have a character device path of
-/dev/spdk/nvmeX, where X is replaced with a unique integer to differentiate it from
-other controllers.  Note that this 'nvmeX' name here has no correlation to the name
-associated with the controller in SPDK.  Namespace character devices will have a path
-of /dev/spdk/nvmeXnY, where Y is the namespace ID.
+This command will register /dev/spdk/nvme0 character device associated with Nvme0
+controller. If there are namespaces created on Nvme0 controller, for each namespace
+device /dev/spdk/nvme0nX is created.
 
 Cuse devices are removed from system, when NVMe controller is detached or unregistered
 with command:

doc/blob.md

@@ -225,7 +225,7 @@ with SPDK API.
 ### Error Handling
 
 Asynchronous Blobstore callbacks all include an error number that should be checked; non-zero values
-indicate an error. Synchronous calls will typically return an error value if applicable.
+indicate and error. Synchronous calls will typically return an error value if applicable.
 
 ### Asynchronous API
 

doc/blobfs.md

@@ -14,30 +14,30 @@ make
 ~~~
 
 Clone the RocksDB repository from the SPDK GitHub fork into a separate directory.
-Make sure you check out the `6.15.fb` branch.
+Make sure you check out the `spdk-v5.14.3` branch.
 
 ~~~{.sh}
 cd ..
-git clone -b 6.15.fb https://github.com/spdk/rocksdb.git
+git clone -b spdk-v5.14.3 https://github.com/spdk/rocksdb.git
 ~~~
 
 Build RocksDB.  Only the `db_bench` benchmarking tool is integrated with BlobFS.
 
 ~~~{.sh}
 cd rocksdb
-make db_bench SPDK_DIR=relative_path/to/spdk
+make db_bench SPDK_DIR=path/to/spdk
 ~~~
 
 Or you can also add `DEBUG_LEVEL=0` for a release build (need to turn on `USE_RTTI`).
 
 ~~~{.sh}
-export USE_RTTI=1 && make db_bench DEBUG_LEVEL=0 SPDK_DIR=relative_path/to/spdk
+export USE_RTTI=1 && make db_bench DEBUG_LEVEL=0 SPDK_DIR=path/to/spdk
 ~~~
 
 Create an NVMe section in the configuration file using SPDK's `gen_nvme.sh` script.
 
 ~~~{.sh}
-scripts/gen_nvme.sh --json-with-subsystems > /usr/local/etc/spdk/rocksdb.json
+scripts/gen_nvme.sh > /usr/local/etc/spdk/rocksdb.json
 ~~~
 
 Verify the configuration file has specified the correct NVMe SSD.

doc/ci_tools.md

@@ -1,6 +0,0 @@
-# CI Tools {#ci_tools}
-
-Section describing tools used by CI to verify integrity of the submitted
-patches ([status](https://ci.spdk.io)).
-
-- @subpage shfmt

doc/general.md

@@ -1,6 +1,5 @@
 # General Information {#general}
 
 - @subpage event
-- @subpage scheduler
 - @subpage logical_volumes
 - @subpage accel_fw

doc/index.md

@@ -1,4 +1,4 @@
-# Storage Performance Development Kit {#mainpage}
+# Storage Performance Development Kit {#index}
 
 # Introduction
 
@@ -32,10 +32,6 @@
 
 @copydoc tools
 
-# CI Tools
-
-@copydoc ci_tools
-
 # Performance Reports
 
 @copydoc performance_reports

doc/intro.md

@@ -4,5 +4,4 @@
 - @subpage getting_started
 - @subpage vagrant
 - @subpage changelog
-- @subpage deprecation
 - [Source Code (GitHub)](https://github.com/spdk/spdk)

doc/libraries.md

@@ -191,23 +191,3 @@ shim/implementation library system.
 	# two libraries
 	gcc -o my_app ./my_app.c -lspdk -lcustom_env_shim -lcustom_env_implementation
 ~~~
-
-# SPDK Static Objects {#static_objects}
-
-SPDK static objects are compiled by default even when no parameters are supplied to the build system.
-Unlike SPDK shared objects, the filename does not contain any versioning semantics. Linking against
-static objects is similar to shared objects but will always require the use of `-Wl,--whole-archive`
-as argument. This is due to the use of constructor functions in SPDK such as those to register
-NVMe transports.
-
-Due to the lack of versioning semantics, it is not recommended to install static libraries system wide.
-Instead the path to these static libraries should be added as argument at compile time using
-`-L/path/to/static/libs`. The use of static objects instead of shared objects can also be forced
-through `-Wl,-Bsatic`, otherwise some compilers might prefer to use the shared objects if both
-are available.
-
-~~~{.sh}
-	gcc -o my_app ./my_app.c -L/path/to/static/libs -Wl,--whole-archive -Wl,-Bstatic -lpassthru_external
-	-lspdk_event_bdev -lspdk_bdev -lspdk_bdev_malloc -lspdk_log -lspdk_thread -lspdk_util -lspdk_event
-	-lspdk_env_dpdk -Wl,--no-whole-archive -Wl,-Bdynamic -pthread -ldpdk
-~~~

doc/nvme-cli.md

@@ -0,0 +1,96 @@
+# nvme-cli {#nvme-cli}
+
+# nvme-cli with SPDK Getting Started Guide
+
+Now nvme-cli can support both kernel driver and SPDK user mode driver for most of its available commands and
+Intel specific commands.
+
+1. Clone the nvme-cli repository from the SPDK GitHub fork. Make sure you check out the spdk-1.6 branch.
+
+  ~~~{.sh}
+  git clone -b spdk-1.6 https://github.com/spdk/nvme-cli.git
+  ~~~
+
+2. Clone the SPDK repository from https://github.com/spdk/spdk under the nvme-cli folder.
+
+3. Refer to the "README.md" under SPDK folder to properly build SPDK.
+
+4. Refer to the "README.md" under nvme-cli folder to properly build nvme-cli.
+
+5. Execute "<spdk_folder>/scripts/setup.sh" with the "root" account.
+
+6. Update the "spdk.conf" file under nvme-cli folder to properly configure the SPDK. Notes as following:
+
+  ~~~{.sh}
+  spdk=1
+  Indicates whether or not to use spdk. Can be 0 (off) or 1 (on).
+  Defaults to 1 which assumes that you have run "<spdk_folder>/scripts/setup.sh", unbinding your drives from the kernel.
+
+  core_mask=0x1
+  A bitmask representing which core(s) to use for nvme-cli operations.
+  Defaults to core 0.
+
+  mem_size=512
+  The amount of reserved hugepage memory to use for nvme-cli (in MB).
+  Defaults to 512MB.
+
+  shm_id=0
+  Indicates the shared memory ID for the spdk application with which your NVMe drives are associated,
+  and should be adjusted accordingly.
+  Defaults to 0.
+~~~
+
+7. Run the "./nvme list" command to get the domain:bus:device.function for each found NVMe SSD.
+
+8. Run the other nvme commands with domain:bus:device.function instead of "/dev/nvmeX" for the specified device.
+
+  ~~~{.sh}
+  Example: ./nvme smart-log 0000:01:00.0
+  ~~~
+
+9. Run the "./nvme intel" commands for Intel specific commands against Intel NVMe SSD.
+
+  ~~~{.sh}
+  Example: ./nvme intel internal-log 0000:08:00.0
+  ~~~
+
+10. Execute "<spdk_folder>/scripts/setup.sh reset" with the "root" account and update "spdk=0" in spdk.conf to
+    use the kernel driver if wanted.
+
+## Use scenarios
+
+### Run as the only SPDK application on the system
+
+1. Modify the spdk to 1 in spdk.conf. If the system has fewer cores or less memory, update the spdk.conf accordingly.
+
+### Run together with other running SPDK applications on shared NVMe SSDs
+
+1. For the other running SPDK application, start with the parameter like "-i 1" to have the same "shm_id".
+
+2. Use the default spdk.conf setting where "shm_id=1" to start the nvme-cli.
+
+3. If other SPDK applications run with different shm_id parameter, update the "spdk.conf" accordingly.
+
+### Run with other running SPDK applications on non-shared NVMe SSDs
+
+1. Properly configure the other running SPDK applications.
+
+  ~~~{.sh}
+  a. Only access the NVMe SSDs it wants.
+  b. Allocate a fixed number of memory instead of all available memory.
+  ~~~
+
+2. Properly configure the spdk.conf setting for nvme-cli.
+
+  ~~~{.sh}
+  a. Not access the NVMe SSDs from other SPDK applications.
+  b. Change the mem_size to a proper size.
+  ~~~
+
+## Note
+
+1. To run the newly built nvme-cli, either explicitly run as "./nvme" or added it into the $PATH to avoid
+   invoke other already installed version.
+
+2. To run the newly built nvme-cli with SPDK support in arbitrary directory, copy "spdk.conf" to that
+   directory from the nvme cli folder and update the configuration as suggested.

doc/nvme.md

@@ -304,8 +304,6 @@ At the NVMe driver level, we provide the following support for Hotplug:
 
 This feature is considered as experimental.
 
-## Design
-
 ![NVMe character devices processing diagram](nvme_cuse.svg)
 
 For each controller as well as namespace, character devices are created in the
@@ -324,76 +322,12 @@ nvme_io_msg_process().
 Ioctls that request information attained when attaching NVMe controller receive an
 immediate response, without passing them through the ring.
 
-This interface reserves one additional qpair for sending down the I/O for each controller.
+This interface reserves one qpair for sending down the I/O for each controller.
 
-## Usage
+## Enabling cuse support for NVMe
 
-### Enabling cuse support for NVMe
-
-Cuse support is disabled by default. To enable support for NVMe-CUSE devices first
-install required dependencies
-~~~{.sh}
-sudo scripts/pkgdep.sh --fuse
-~~~
-Then compile SPDK with "./configure --with-nvme-cuse".
-
-### Creating NVMe-CUSE device
-
-First make sure to prepare the environment (see @ref getting_started).
-This includes loading CUSE kernel module.
-Any NVMe controller attached to a running SPDK application can be
-exposed via NVMe-CUSE interface. When closing SPDK application,
-the NVMe-CUSE devices are unregistered.
-
-~~~{.sh}
-$ sudo scripts/setup.sh
-$ sudo modprobe cuse
-$ sudo build/bin/spdk_tgt
-# Continue in another session
-$ sudo scripts/rpc.py bdev_nvme_attach_controller -b Nvme0 -t PCIe -a 0000:82:00.0
-Nvme0n1
-$ sudo scripts/rpc.py bdev_nvme_get_controllers
-[
-  {
-    "name": "Nvme0",
-    "trid": {
-      "trtype": "PCIe",
-      "traddr": "0000:82:00.0"
-    }
-  }
-]
-$ sudo scripts/rpc.py bdev_nvme_cuse_register -n Nvme0
-$ ls /dev/spdk/
-nvme0  nvme0n1
-~~~
-
-### Example of using nvme-cli
-
-Most nvme-cli commands can point to specific controller or namespace by providing a path to it.
-This can be leveraged to issue commands to the SPDK NVMe-CUSE devices.
-
-~~~{.sh}
-sudo nvme id-ctrl /dev/spdk/nvme0
-sudo nvme smart-log /dev/spdk/nvme0
-sudo nvme id-ns /dev/spdk/nvme0n1
-~~~
-
-Note: `nvme list` command does not display SPDK NVMe-CUSE devices,
-see nvme-cli [PR #773](https://github.com/linux-nvme/nvme-cli/pull/773).
-
-### Examples of using smartctl
-
-smartctl tool recognizes device type based on the device path. If none of expected
-patterns match, SCSI translation layer is used to identify device.
-
-To use smartctl '-d nvme' parameter must be used in addition to full path to
-the NVMe device.
-
-~~~{.sh}
-    smartctl -d nvme -i /dev/spdk/nvme0
-    smartctl -d nvme -H /dev/spdk/nvme1
-    ...
-~~~
+Cuse support is disabled by default. To enable support for NVMe devices SPDK
+must be compiled with "./configure --with-nvme-cuse".
 
 ## Limitations
 
@@ -408,3 +342,17 @@ with SPDK NVMe CUSE.
 
 SCSI to NVMe Translation Layer is not implemented. Tools that are using this layer to
 identify, manage or operate device might not work properly or their use may be limited.
+
+### Examples of using smartctl
+
+smartctl tool recognizes device type based on the device path. If none of expected
+patterns match, SCSI translation layer is used to identify device.
+
+To use smartctl '-d nvme' parameter must be used in addition to full path to
+the NVMe device.
+
+~~~{.sh}
+    smartctl -d nvme -i /dev/spdk/nvme0
+    smartctl -d nvme -H /dev/spdk/nvme1
+    ...
+~~~

doc/nvme_spec.md

@@ -20,8 +20,8 @@ registers involved that are called doorbells.
 
 An I/O is submitted to an NVMe device by constructing a 64 byte command, placing
 it into the submission queue at the current location of the submission queue
-tail index, and then writing the new index of the submission queue tail to the
-submission queue tail doorbell register. It's actually valid to copy a whole set
+head index, and then writing the new index of the submission queue head to the
+submission queue head doorbell register. It's actually valid to copy a whole set
 of commands into open slots in the ring and then write the doorbell just one
 time to submit the whole batch.
 

doc/nvmf.md

@@ -106,14 +106,20 @@ using 1GB hugepages or by pre-reserving memory at application startup with `--me
 option. All pre-reserved memory will be registered as a single region, but won't be returned to the
 system until the SPDK application is terminated.
 
-Another known issue occurs when using the E810 NICs in RoCE mode. Specifically, the NVMe-oF target
-sometimes cannot destroy a qpair, because its posted work requests don't get flushed.  It can cause
-the NVMe-oF target application unable to terminate cleanly.
-
 ## TCP transport support {#nvmf_tcp_transport}
 
 The transport is built into the nvmf_tgt by default, and it does not need any special libraries.
 
+## Configuring the SPDK NVMe over Fabrics Target {#nvmf_config}
+
+An NVMe over Fabrics target can be configured using JSON RPCs.
+The basic RPCs needed to configure the NVMe-oF subsystem are detailed below. More information about
+working with NVMe over Fabrics specific RPCs can be found on the @ref jsonrpc_components_nvmf_tgt RPC page.
+
+Using .ini style configuration files for configuration of the NVMe-oF target is deprecated and should
+be replaced with JSON based RPCs. .ini style configuration files can be converted to json format by way
+of the new script `scripts/config_converter.py`.
+
 ## FC transport support {#nvmf_fc_transport}
 
 To build nvmf_tgt with the FC transport, there is an additional FC LLD (Low Level Driver) code dependency.
@@ -141,12 +147,6 @@ cd ../spdk
 make
 ~~~
 
-## Configuring the SPDK NVMe over Fabrics Target {#nvmf_config}
-
-An NVMe over Fabrics target can be configured using JSON RPCs.
-The basic RPCs needed to configure the NVMe-oF subsystem are detailed below. More information about
-working with NVMe over Fabrics specific RPCs can be found on the @ref jsonrpc_components_nvmf_tgt RPC page.
-
 ### Using RPCs {#nvmf_config_rpc}
 
 Start the nvmf_tgt application with elevated privileges. Once the target is started,

doc/nvmf_tgt_pg.md

@@ -68,7 +68,7 @@ system. This is used for access control.
 
 A user of the NVMe-oF target library begins by creating a target using
 spdk_nvmf_tgt_create(), setting up a set of addresses on which to accept
-connections by calling spdk_nvmf_tgt_listen_ext(), then creating a subsystem
+connections by calling spdk_nvmf_tgt_listen(), then creating a subsystem
 using spdk_nvmf_subsystem_create().
 
 Subsystems begin in an inactive state and must be activated by calling
@@ -78,7 +78,7 @@ calling spdk_nvmf_subsystem_pause() and resumed by calling
 spdk_nvmf_subsystem_resume().
 
 Namespaces may be added to the subsystem by calling
-spdk_nvmf_subsystem_add_ns_ext() when the subsystem is inactive or paused.
+spdk_nvmf_subsystem_add_ns() when the subsystem is inactive or paused.
 Namespaces are bdevs. See @ref bdev for more information about the SPDK bdev
 layer. A bdev may be obtained by calling spdk_bdev_get_by_name().
 

doc/nvmf_tracing.md

@@ -202,4 +202,4 @@ record the current trace state of several tracepoints.
 		...
 ~~~
 
-All the tracing functions are documented in the [Tracepoint library documentation](https://spdk.io/doc/trace_8h.html)
+All the tracing functions are documented in the [Tracepoint library documentation](https://www.spdk.io/doc/trace_8h.html)

doc/performance_reports.md

@@ -1,19 +1,5 @@
 # Performance Reports {#performance_reports}
 
-## Release 21.01
-
-- [SPDK 21.01 NVMe Bdev Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_nvme_bdev_perf_report_2101.pdf)
-- [SPDK 21.01 NVMe-oF TCP Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_tcp_perf_report_2101.pdf)
-- [SPDK 21.01 NVMe-oF RDMA Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_rdma_perf_report_2101.pdf)
-- [SPDK 21.01 Vhost Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_vhost_perf_report_2101.pdf)
-
-## Release 20.10
-
-- [SPDK 20.10 NVMe Bdev Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_nvme_bdev_perf_report_2010.pdf)
-- [SPDK 20.10 NVMe-oF TCP Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_tcp_perf_report_2010.pdf)
-- [SPDK 20.10 NVMe-oF RDMA Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_rdma_perf_report_2010.pdf)
-- [SPDK 20.10 Vhost Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_vhost_perf_report_2010.pdf)
-
 ## Release 20.07
 
 - [SPDK 20.07 NVMe-oF TCP Performance Report](https://ci.spdk.io/download/performance-reports/SPDK_tcp_perf_report_2007.pdf)

doc/pkgconfig.md

@@ -1,56 +0,0 @@
-# Linking SPDK applications with pkg-config {#pkgconfig}
-
-The SPDK build system generates pkg-config files to facilitate linking
-applications with the correct set of SPDK and DPDK libraries. Using pkg-config
-in your build system will ensure you do not need to make modifications
-when SPDK adds or modifies library dependencies.
-
-If your application is using the SPDK nvme library, you would use the following
-to get the list of required SPDK libraries:
-
-~~~
-PKG_CONFIG_PATH=/path/to/spdk/build/lib/pkgconfig pkg-config --libs spdk_nvme
-~~~
-
-To get the list of required SPDK and DPDK libraries to use the DPDK-based
-environment layer:
-
-~~~
-PKG_CONFIG_PATH=/path/to/spdk/build/lib/pkgconfig pkg-config --libs spdk_env_dpdk
-~~~
-
-When linking with static libraries, the dependent system libraries must also be
-specified. To get the list of required system libraries:
-
-~~~
-PKG_CONFIG_PATH=/path/to/spdk/build/lib/pkgconfig pkg-config --libs spdk_syslibs
-~~~
-
-Note that SPDK libraries use constructor functions liberally, so you must surround
-the library list with extra linker options to ensure these functions are not dropped
-from the resulting application binary. With shared libraries this is achieved through
-the `-Wl,--no-as-needed` parameters while with static libraries `-Wl,--whole-archive`
-is used. Here is an example Makefile snippet that shows how to use pkg-config to link
-an application that uses the SPDK nvme shared library:
-
-~~~
-PKG_CONFIG_PATH = $(SPDK_DIR)/build/lib/pkgconfig
-SPDK_LIB := $(shell PKG_CONFIG_PATH="$(PKG_CONFIG_PATH)" pkg-config --libs spdk_nvme
-DPDK_LIB := $(shell PKG_CONFIG_PATH="$(PKG_CONFIG_PATH)" pkg-config --libs spdk_env_dpdk
-
-app:
-	$(CC) -o app app.o -pthread -Wl,--no-as-needed $(SPDK_LIB) $(DPDK_LIB) -Wl,--as-needed
-~~~
-
-If using the SPDK nvme static library:
-
-~~~
-PKG_CONFIG_PATH = $(SPDK_DIR)/build/lib/pkgconfig
-SPDK_LIB := $(shell PKG_CONFIG_PATH="$(PKG_CONFIG_PATH)" pkg-config --libs spdk_nvme
-DPDK_LIB := $(shell PKG_CONFIG_PATH="$(PKG_CONFIG_PATH)" pkg-config --libs spdk_env_dpdk
-SYS_LIB := $(shell PKG_CONFIG_PATH="$(PKG_CONFIG_PATH)" pkg-config --libs --static spdk_syslibs
-
-app:
-	$(CC) -o app app.o -pthread -Wl,--whole-archive $(SPDK_LIB) $(DPDK_LIB) -Wl,--no-whole-archive \
-		$(SYS_LIB)
-~~~

doc/rpm.md

@@ -1,49 +0,0 @@
-# RPMs {#rpms}
-
-# In this document {#rpms_toc}
-
-* @ref building_rpms
-
-# Building SPDK RPMs {#building_rpms}
-
-To build basic set of RPM packages out of the SPDK repo simply run:
-
-~~~{.sh}
-# rpmbuild/rpm.sh
-~~~
-
-Additional configuration options can be passed directly as arguments:
-
-~~~{.sh}
-# rpmbuild/rpm.sh --with-shared --with-dpdk=/path/to/dpdk/build
-~~~
-
-There are several options that may be passed via environment as well:
-
-- DEPS          - Install all needed dependencies for building RPM packages.
-                Default: "yes"
-- MAKEFLAGS     - Flags passed to make
-- RPM_RELEASE   - Target release version of the RPM packages. Default: 1
-- REQUIREMENTS  - Extra set of RPM dependencies if deemed as needed
-- SPDK_VERSION  - SPDK version. Default: currently checked out tag
-
-~~~{.sh}
-# DEPS=no MAKEFLAGS="-d -j1" rpmbuild/rpm.sh --with-shared
-~~~
-
-By default, all RPM packages should be created under $HOME directory of the
-target user:
-
-~~~{.sh}
-# printf '%s\n' /root/rpmbuild/RPMS/x86_64/*
-/root/rpmbuild/RPMS/x86_64/spdk-devel-v21.01-1.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/spdk-dpdk-libs-v21.01-1.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/spdk-libs-v21.01-1.x86_64.rpm
-/root/rpmbuild/RPMS/x86_64/spdk-v21.01-1.x86_64.rpm
-#
-~~~
-
-- spdk            - provides all the binaries, common tooling, etc.
-- spdk-devel      - provides development files
-- spdk-libs       - provides target lib, .pc files (--with-shared)
-- spdk-dpdk-libs  - provides dpdk lib files (--with-shared|--with-dpdk)

doc/scheduler.md

@@ -1,82 +0,0 @@
-# Scheduler {#scheduler}
-
-SPDK's event/application framework (`lib/event`) now supports scheduling of
-lightweight threads. Schedulers are provided as plugins, called
-implementations. A default implementation is provided, but users may wish to
-write their own scheduler to integrate into broader code frameworks or meet
-their performance needs.
-
-This feature should be considered experimental and is disabled by default. When
-enabled, the scheduler framework gathers data for each spdk thread and reactor
-and passes it to a scheduler implementation to perform one of the following
-actions.
-
-## Actions
-
-### Move a thread
-
-`spdk_thread`s can be moved to another reactor. Schedulers can examine the
-suggested cpu_mask value for each lightweight thread to see if the user has
-requested specific reactors, or choose a reactor using whatever algorithm they
-deem fit.
-
-### Switch reactor mode
-
-Reactors by default run in a mode that constantly polls for new actions for the
-most efficient processing. Schedulers can switch a reactor into a mode that
-instead waits for an event on a file descriptor. On Linux, this is implemented
-using epoll. This results in reduced CPU usage but may be less responsive when
-events occur. A reactor cannot enter this mode if any `spdk_threads` are
-currently scheduled to it. This limitation is expected to be lifted in the
-future, allowing `spdk_threads` to enter interrupt mode.
-
-### Set frequency of CPU core
-
-The frequency of CPU cores can be modified by the scheduler in response to
-load. Only CPU cores that match the application cpu_mask may be modified. The
-mechanism for controlling CPU frequency is pluggable and the default provided
-implementation is called `dpdk_governor`, based on the `rte_power` library from
-DPDK.
-
-#### Known limitation
-
-When SMT (Hyperthreading) is enabled the two logical CPU cores sharing a single
-physical CPU core must run at the same frequency. If one of two of such logical
-CPU cores is outside the application cpu_mask, the policy and frequency on that
-core has to be managed by the administrator.
-
-## Scheduler implementations
-
-The scheduler in use may be controlled by JSON-RPC. Please use the
-[framework_set_scheduler](jsonrpc.md/#rpc_framework_set_scheduler) RPC to
-switch between schedulers or change their options.
-
-[spdk_top](spdk_top.md#spdk_top) is a useful tool to observe the behavior of
-schedulers in different scenarios and workloads.
-
-### static [default]
-
-The `static` scheduler is the default scheduler and does no dynamic scheduling.
-Lightweight threads are distributed round-robin among reactors, respecting
-their requested cpu_mask, and then they are never moved. This is equivalent to
-the previous behavior of the SPDK event/application framework.
-
-### dynamic
-
-The `dynamic` scheduler is designed for power saving and reduction of CPU
-utilization, especially in cases where workloads show large variations over
-time.
-
-Active threads are distributed equally among reactors, taking cpu_mask into
-account. All idle threads are moved to the main core. Once an idle thread becomes
-active, it is redistributed again.
-
-When a reactor has no scheduled `spdk_thread`s it is switched into interrupt
-mode and stops actively polling. After enough threads become active, the
-reactor is switched back into poll mode and threads are assigned to it again.
-
-The main core can contain active threads only when their execution time does
-not exceed the sum of all idle threads. When no active threads are present on
-the main core, the frequency of that CPU core will decrease as the load
-decreases. All CPU cores corresponding to the other reactors remain at maximum
-frequency.

doc/shfmt.md

@@ -1,146 +0,0 @@
-# shfmt {#shfmt}
-
-# In this document {#shfmt_toc}
-
-* @ref shfmt_overview
-* @ref shfmt_usage
-* @ref shfmt_installation
-* @ref shfmt_examples
-
-# Overview {#shfmt_overview}
-
-The majority of tests (and scripts overall) in the SPDK repo are written
-in Bash (with a quite significant emphasis on "Bashism"), thus a style
-formatter, shfmt, was introduced to help keep the .sh code consistent
-across the entire repo. For more details on the tool itself, please see
-[shfmt](https://github.com/mvdan/sh).
-
-# Usage {#shfmt_usage}
-
-On the CI pool, the shfmt is run against all the updated .sh files that
-have been committed but not merged yet. Additionally, shfmt will pick
-all .sh present in the staging area when run locally from our pre-commit
-hook (via check_format.sh). In case any style errors are detected, a
-patch with needed changes is going to be generated and either build (CI)
-or the commit will be aborted. Said patch can be then easily applied:
-
-~~~{.sh}
-# Run from the root of the SPDK repo
-patch --merge -p0 <shfmt-3.1.0.patch
-~~~
-
-The name of the patch is derived from the version of shfmt that is
-currently in use (3.1.0 is currently supported).
-
-Please, see ./scripts/check_format.sh for all the arguments the shfmt
-is run with. Additionally, @ref shfmt_examples has more details on how
-each of the arguments behave.
-
-# Installation {#shfmt_installation}
-
-The shfmt can be easily installed via pkgdep.sh:
-
-~~~{.sh}
-./scripts/pkgdep.sh -d
-~~~
-
-This will install all the developers tools, including shfmt, on the
-local system. The precompiled binary will be saved, by default, to
-/opt/shfmt and then linked under /usr/bin. Both paths can be changed
-by setting SHFMT_DIR and SHFMT_DIR_OUT in the environment. Example:
-
-~~~{.sh}
-SHFMT_DIR=/keep_the_binary_here \
-SHFMT_DIR_OUT=/and_link_it_here \
-  ./scripts/pkgdep.sh -d
-~~~
-
-# Examples {#shfmt_examples}
-
-~~~{.sh}
-#######################################
-if foo=$(bar); then
-  echo "$foo"
-fi
-
-exec "$foo" \
-  --bar \
-  --foo
-
-# indent_style = tab
-
-if foo=$(bar); then
-        echo "$foo"
-fi
-
-exec foobar \
-        --bar \
-        --foo
-######################################
-if foo=$(bar); then
-        echo "$foo" && \
-        echo "$(bar)"
-fi
-# binary_next_line = true
-if foo=$(bar); then
-        echo "$foo" \
-                && echo "$(bar)"
-fi
-
-# Note that each break line is also being indented:
-
-if [[ -v foo ]] \
-&& [[ -v bar ]] \
-&& [[ -v foobar ]]; then
-	echo "This is foo"
-fi
-# ->
-if [[ -v foo ]] \
-        && [[ -v bar ]] \
-        && [[ -v foobar ]]; then
-        echo "This is foo"
-fi
-
-# Currently, newlines are being escaped even if syntax-wise
-# they are not needed, thus watch for the following:
-if [[ -v foo
-        && -v bar
-        && -v foobar ]]; then
-        echo "This is foo"
-fi
-#->
-if [[ -v foo && -v \
-        bar && -v \
-        foobar ]]; then
-        echo "This is foo"
-fi
-# This, unfortunately, also breaks the -bn behavior.
-# (see https://github.com/mvdan/sh/issues/565) for details.
-######################################
-case "$FOO" in
-        BAR)
-        echo "$FOO" ;;
-esac
-# switch_case_indent = true
-case "$FOO" in
-        BAR)
-                echo "$FOO"
-                ;;
-esac
-######################################
-exec {foo}>bar
-:>foo
-exec {bar}<foo
-# -sr
-exec {foo}> bar
-: > foo
-exec {bar}< foo
-######################################
-# miscellaneous, enforced by shfmt
-(( no_spacing_at_the_beginning & ~and_no_spacing_at_the_end ))
-: $(( no_spacing_at_the_beginning & ~and_no_spacing_at_the_end ))
-
-# ->
-((no_spacing_at_the_beginning & ~and_no_spacing_at_the_end))
-: $((no_spacing_at_the_beginning & ~and_no_spacing_at_the_end))
-~~~

doc/spdk_top.md

@@ -5,7 +5,7 @@ The spdk_top application is designed to resemble the standard top in that it pro
 Why doesn't the classic top utility work for SPDK? SPDK uses a polled-mode design; a reactor thread running on each CPU core assigned to an SPDK application schedules SPDK lightweight threads and pollers to run on the CPU core. Therefore, the standard Linux top utility is not effective for analyzing the CPU usage for polled-mode applications like SPDK because it just reports that they are using 100% of the CPU resources assigned to them. The spdk_top utility was developed to analyze and report the CPU cycles used to do real work vs just polling for work. The utility relies on instrumentation added to pollers to track when they are doing work vs. polling for work. The spdk_top utility gets the fine grained metrics from the pollers, analyzes and report the metrics on a per poller, thread and core basis. This information enables users to identify CPU cores that are busy doing real work so that they can determine if the application needs more or less CPU resources.
 
 # Run spdk_top
-Before running spdk_top you need to run the SPDK application whose performance you want to analyze using spdk_top.
+Before running spdk_top you need to run the SPDK application whose performance you want to analyze using spdk_top. For example, the nvmf_tgt application was running when we used the spdk_top to capture the screen shots in this documentation.
 
 Run the spdk_top application
 
@@ -13,53 +13,33 @@ Run the spdk_top application
 ./build/bin/spdk_top
 ~~~
 
-# Bottom menu
-Menu at the bottom of SPDK top window shows many options for changing displayed data. Each menu item has a key associated with it in square brackets.
-
-* Quit - quits the SPDK top application.
-* TAB selection - allows to select THREADS/POLLERS/CORES tabs.
-* Previous page/Next page - scrolls up/down to the next set of rows displayed. Indicator in the bottom-left corner shows current page and number of all available pages.
-* Columns - enables/disables chosen columns in a column pop-up window.
-* Sorting - allows to sort displayed data by column in a sorting pop-up.
-* Refresh rate - takes user input from 0 to 255 and changes refresh rate to that value in seconds.
-* Item details - displays details pop-up window for highlighted data row. Selection is changed by pressing UP and DOWN arrow keys.
-* Total/Interval - changes displayed values in all tabs to either Total time (measured since start of SPDK application) or Interval time (measured since last refresh).
+The spdk_top application has 3 tabs: the cores, threads and pollers tabs.
 
 # Threads Tab
-The threads tab displays a line item for each spdk thread. The information displayed shows:
+The threads tab displays a line item for each spdk thread that includes information such as which CPU core the spdk thread is running on, how many pollers the thread is running and how many microseconds was the thread busy/idle. The pollers are grouped into active, timed and pause pollers. To learn more about spdk threads see @ref concurrency.
 
-* Thread name - name of SPDK thread.
-* Core - core on which the thread is currently running.
-* Active/Timed/Paused pollers - number of pollers grouped by type on this thread.
-* Idle/Busy - how many microseconds the thread was idle/busy.
-
-\n
-By pressing ENTER key a pop-up window appears, showing above and a list of pollers running on selected thread (with poller name, type, run count and period).
-Pop-up then can be closed by pressing ESC key.
-
-To learn more about spdk threads see @ref concurrency.
+![Threads Tab](img/spdk_top_page1_threads.png)
 
 # Pollers Tab
-The pollers tab displays a line item for each poller. The information displayed shows:
+The pollers tab displays a line item for each poller and a running counter of the number of times the poller has run so that you can see which pollers are running most frequently.
 
-* Poller name - name of currently selected poller.
-* Type - type of poller (Active/Paused/Timed).
-* On thread - thread on which the poller is running.
-* Run count - how many times poller was run.
-* Period - poller period in microseconds. If period equals 0 then it is not displayed.
-* Status - whether poller is currently Busy (red color) or Idle (blue color).
-
-\n
-Poller pop-up window can be displayed by pressing ENTER on a selected data row and displays above information.
-Pop-up can be closed by pressing ESC key.
+![Pollers Tab](img/spdk_top_page2_pollers.png)
 
 # Cores Tab
-The cores tab provides insights into how the application is using the CPU cores assigned to it. The information displayed for each core shows:
+The cores tab provides insights into how the application is using the CPU cores assigned to it.
+It has a line item for each CPU core assigned to the application which shows the number of threads and poller
+running on the CPU core. The tab also indicates how busy/idle the each CPU core was in the last 1 second.
+The busy column displays how many microseconds the CPU core was doing actual work in the last 1 second.
+The idle column displays how many microseconds the CPU core was idle in the last 1 second,
+including the time when the CPU core ran pollers but did not find any work.
 
-* Core - core number.
-* Thread count - number of threads currently running on core.
-* Poller count - total number of pollers running on core.
-* Idle/Busy - how many microseconds core was idle (including time when core ran pollers but did not find any work) or doing actual work.
+![Cores Tab](img/spdk_top_page3_cores.png)
 
-\n
-Pressing ENTER key makes a pop-up window appear, showing above information, along with a list of threads running on selected core. Cores details window allows to select a thread and display thread details pop-up on top of it. To close both pop-ups use ESC key.
+# Refresh Rate
+You can control how often the spdk_top application refreshes the data displayed by hitting the 'r' key on your keyboard and specifying a value between 0 and 255 seconds.
+
+# Sorting
+You can sort the data displayed by hitting the 's' key on your keyboard and selecting a column to sort by in the sub menu that is displayed.
+
+# Filtering
+You can filter out any column by hitting the 'c' key on your keyboard and unselecting the column in the menu that is displayed.

doc/system_configuration.md

@@ -18,8 +18,10 @@ x86_64 system, or add `iommu.passthrough=1` on arm64 systems.
 
 There are also some instances where a user may not want to use `uio_pci_generic` or the kernel
 version they are using has a bug where `uio_pci_generic` [fails to bind to NVMe drives](https://github.com/spdk/spdk/issues/399).
-In these cases, users can build the `igb_uio` kernel module which can be found in dpdk-kmods repository.
-To ensure that the driver is properly bound, users should specify `DRIVER_OVERRIDE=/path/to/igb_uio.ko`.
+In these cases, users building with the DPDK submodule can build the `igb_uio` kernel module by
+supplying `--with-igb-uio-driver` to `./configure`. Upon a successful make, the file will be
+located at `dpdk/build/build/kmod/igb_uio.ko`. To ensure that the driver is properly bound, users
+should specify `DRIVER_OVERRIDE=/path/to/igb_uio.ko`.
 
 # Running SPDK as non-priviledged user {#system_configuration_nonroot}
 

doc/tools.md

@@ -1,5 +1,6 @@
 # Tools {#tools}
 
 - @subpage spdkcli
+- @subpage nvme-cli
 - @subpage bdevperf
 - @subpage spdk_top

doc/user_guides.md

@@ -2,7 +2,6 @@
 
 - @subpage system_configuration
 - @subpage libraries
-- @subpage pkgconfig
 - @subpage app_overview
 - @subpage iscsi
 - @subpage nvmf

doc/vhost.md

@@ -222,7 +222,7 @@ host:~# HUGEMEM=2048 ./scripts/setup.sh
 ~~~{.sh}
 host:~# ./build/bin/vhost -S /var/tmp -s 1024 -m 0x3 &
 Starting DPDK 17.11.0 initialization...
-[ DPDK EAL parameters: vhost -c 3 -m 1024 --main-lcore=1 --file-prefix=spdk_pid156014 ]
+[ DPDK EAL parameters: vhost -c 3 -m 1024 --master-lcore=1 --file-prefix=spdk_pid156014 ]
 EAL: Detected 48 lcore(s)
 EAL: Probing VFIO support...
 EAL: VFIO support initialized

doc/vmd.md

@@ -47,11 +47,11 @@ $ 5d:05.5 RAID bus controller: Intel Corporation Device 201d (rev 04)
 $ d7:05.5 RAID bus controller: Intel Corporation Device 201d (rev 04)
 ```
 
-Run setup.sh script with VMD devices set in PCI_ALLOWED.
+Run setup.sh script with VMD devices set in PCI_WHITELIST.
 
 Example:
 ```
-$ PCI_ALLOWED="0000:5d:05.5 0000:d7:05.5" scripts/setup.sh
+$ PCI_WHITELIST="0000:5d:05.5 0000:d7:05.5" scripts/setup.sh
 ```
 
 Check for available devices behind the VMD with spdk_lspci.

dpdk

@@ -1 +1 @@
-Subproject commit 4f93dbc0c0ab3804abaa20123030ad7fccf78709
+Subproject commit 7d8b8e4efe4833631f9a03f18d14e7c642927b8b

dpdkbuild/Makefile

@@ -40,15 +40,14 @@ DPDK_OPTS = -Denable_docs=false
 DPDK_CFLAGS =
 
 DPDK_KMODS = false
+ifeq ($(CONFIG_IGB_UIO_DRIVER),y)
+DPDK_KMODS = true
+endif
 ifeq ($(OS),FreeBSD)
 DPDK_KMODS = true
 endif
 DPDK_OPTS += -Denable_kmods=$(DPDK_KMODS)
 
-ifeq ($(CONFIG_DEBUG),y)
-DPDK_OPTS += --buildtype=debug
-endif
-
 # the drivers we use
 DPDK_DRIVERS = bus bus/pci bus/vdev mempool/ring
 
@@ -73,27 +72,22 @@ endif
 DPDK_OPTS += -Dmachine=$(TARGET_ARCHITECTURE)
 
 ifneq ($(CONFIG_CROSS_PREFIX),)
-ifeq ($(findstring mingw,$(CONFIG_CROSS_PREFIX)),mingw)
-DPDK_OPTS += --cross-file $(SPDK_ROOT_DIR)/dpdk/config/x86/cross-mingw
-else
 $(error Automatic DPDK cross build is not supported. Please compile DPDK manually \
 with e.g. `meson build --cross-file config/arm/arm64_armv8_linux_gcc`)
 endif
-endif
 
 DPDK_CFLAGS += -fPIC
 
+ifeq ($(CONFIG_DEBUG),y)
+DPDK_CFLAGS += -O0 -g
+endif
+
 ifeq ($(CONFIG_WERROR),y)
 DPDK_CFLAGS += -Werror
 else
 DPDK_CFLAGS += -Wno-error
 endif
 
-ifeq ($(CONFIG_CET),y)
-DPDK_CFLAGS += -fcf-protection
-DPDK_LDFLAGS += -fcf-protection
-endif
-
 ifdef EXTRA_DPDK_CFLAGS
 $(warning EXTRA_DPDK_CFLAGS defined, possibly to work around an unsupported compiler version)
 $(shell sleep 1)
@@ -121,7 +115,7 @@ DPDK_ALL_DRIVER_DIRS = $(shell find $(SPDK_ROOT_DIR)/dpdk/drivers -mindepth 1 -t
 DPDK_ALL_DRIVERS = $(DPDK_ALL_DRIVER_DIRS:$(SPDK_ROOT_DIR)/dpdk/drivers/%=%)
 DPDK_DISABLED_DRVERS = $(filter-out $(DPDK_DRIVERS),$(DPDK_ALL_DRIVERS))
 
-ifneq ($(OS),FreeBSD)
+ifeq ($(OS),Linux)
 SED_INPLACE_FLAG = "-i"
 MESON_PREFIX = $(SPDK_ROOT_DIR)/dpdk/build
 else
@@ -137,7 +131,7 @@ ifeq ($(MAKE_PID),)
 MAKE_PID := $(shell echo $$PPID)
 endif
 
-MAKE_NUMJOBS := $(shell ps T | sed -nE 's/[[:space:]]*$(MAKE_PID)[[:space:]].* (-j|--jobs=)( *[0-9]+).*/\1\2/p')
+MAKE_NUMJOBS := $(shell ps T | sed -nE 's/\s*$(MAKE_PID)\s.* (-j|--jobs=)( *[0-9]+).*/\1\2/p')
 
 all: $(SPDK_ROOT_DIR)/dpdk/build-tmp
 	$(Q)# DPDK doesn't handle nested make calls, so unset MAKEFLAGS

examples/Makefile

@@ -37,7 +37,7 @@ include $(SPDK_ROOT_DIR)/mk/spdk.common.mk
 DIRS-y += accel bdev blob ioat nvme sock vmd nvmf
 
 ifeq ($(OS),Linux)
-DIRS-$(CONFIG_VHOST) += interrupt_tgt
+DIRS-y += interrupt_tgt
 endif
 
 .PHONY: all clean $(DIRS-y)

examples/accel/perf/Makefile

@@ -39,6 +39,9 @@ APP = accel_perf
 
 C_SRCS := accel_perf.c
 
-SPDK_LIB_LIST = $(ACCEL_MODULES_LIST) event_accel
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM)
+SPDK_LIB_LIST += bdev accel event thread util conf trace \
+		log jsonrpc json rpc sock notify
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/bdev/fio_plugin/Makefile

@@ -40,6 +40,8 @@ FIO_PLUGIN := spdk_bdev
 
 C_SRCS = fio_plugin.c
 
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_bdev
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += thread util bdev conf accel rpc jsonrpc json log sock trace notify
+SPDK_LIB_LIST += event $(EVENT_BDEV_SUBSYSTEM)
 
 include $(SPDK_ROOT_DIR)/mk/spdk.fio.mk

examples/bdev/fio_plugin/README.md

@@ -60,10 +60,8 @@ Or for NVMe devices:
 
     filename=Nvme0n1
 
-fio by default forks a separate process for every job. It also supports just spawning a separate
-thread in the same process for every job. The SPDK fio plugin is limited to this latter thread
-usage model, so fio jobs must also specify thread=1 when using the SPDK fio plugin. The SPDK fio
-plugin supports multiple threads - in this case, the "1" just means "use thread mode".
+Currently the SPDK fio plugin is limited to the thread usage model, so fio jobs must also specify thread=1
+when using the SPDK fio plugin.
 
 fio also currently has a race condition on shutdown if dynamically loading the ioengine by specifying the
 engine's full path via the ioengine parameter - LD_PRELOAD is recommended to avoid this race condition.

examples/bdev/fio_plugin/fio_plugin.c

@@ -42,6 +42,7 @@
 #include "spdk/queue.h"
 #include "spdk/util.h"
 
+#include "spdk_internal/thread.h"
 #include "spdk_internal/event.h"
 
 #include "config-host.h"
@@ -59,7 +60,7 @@ struct spdk_fio_options {
 	char *conf;
 	char *json_conf;
 	unsigned mem_mb;
-	int mem_single_seg;
+	bool mem_single_seg;
 };
 
 struct spdk_fio_request {
@@ -675,11 +676,6 @@ spdk_fio_queue(struct thread_data *td, struct io_u *io_u)
 				     io_u->offset, io_u->xfer_buflen,
 				     spdk_fio_completion_cb, fio_req);
 		break;
-	case DDIR_SYNC:
-		rc = spdk_bdev_flush(target->desc, target->ch,
-				     io_u->offset, io_u->xfer_buflen,
-				     spdk_fio_completion_cb, fio_req);
-		break;
 	default:
 		assert(false);
 		break;
@@ -789,7 +785,6 @@ static struct fio_option options[] = {
 		.type		= FIO_OPT_BOOL,
 		.off1		= offsetof(struct spdk_fio_options, mem_single_seg),
 		.help		= "If set to 1, SPDK will use just a single hugetlbfs file",
-		.def            = "0",
 		.category	= FIO_OPT_C_ENGINE,
 		.group		= FIO_OPT_G_INVALID,
 	},

examples/bdev/hello_world/Makefile

@@ -37,6 +37,8 @@ APP = hello_bdev
 
 C_SRCS := hello_bdev.c
 
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_bdev
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM)
+SPDK_LIB_LIST += bdev accel event thread util conf trace log jsonrpc json rpc sock notify
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/bdev/hello_world/hello_bdev.c

@@ -38,7 +38,7 @@
 #include "spdk/event.h"
 #include "spdk/log.h"
 #include "spdk/string.h"
-#include "spdk/bdev_zone.h"
+#include "spdk/bdev_module.h"
 
 static char *g_bdev_name = "Malloc0";
 
@@ -191,50 +191,6 @@ hello_bdev_event_cb(enum spdk_bdev_event_type type, struct spdk_bdev *bdev,
 	SPDK_NOTICELOG("Unsupported bdev event: type %d\n", type);
 }
 
-static void
-reset_zone_complete(struct spdk_bdev_io *bdev_io, bool success, void *cb_arg)
-{
-	struct hello_context_t *hello_context = cb_arg;
-
-	/* Complete the I/O */
-	spdk_bdev_free_io(bdev_io);
-
-	if (!success) {
-		SPDK_ERRLOG("bdev io reset zone error: %d\n", EIO);
-		spdk_put_io_channel(hello_context->bdev_io_channel);
-		spdk_bdev_close(hello_context->bdev_desc);
-		spdk_app_stop(-1);
-		return;
-	}
-
-	hello_write(hello_context);
-}
-
-static void
-hello_reset_zone(void *arg)
-{
-	struct hello_context_t *hello_context = arg;
-	int rc = 0;
-
-	rc = spdk_bdev_zone_management(hello_context->bdev_desc, hello_context->bdev_io_channel,
-				       0, SPDK_BDEV_ZONE_RESET, reset_zone_complete, hello_context);
-
-	if (rc == -ENOMEM) {
-		SPDK_NOTICELOG("Queueing io\n");
-		/* In case we cannot perform I/O now, queue I/O */
-		hello_context->bdev_io_wait.bdev = hello_context->bdev;
-		hello_context->bdev_io_wait.cb_fn = hello_reset_zone;
-		hello_context->bdev_io_wait.cb_arg = hello_context;
-		spdk_bdev_queue_io_wait(hello_context->bdev, hello_context->bdev_io_channel,
-					&hello_context->bdev_io_wait);
-	} else if (rc) {
-		SPDK_ERRLOG("%s error while resetting zone: %d\n", spdk_strerror(-rc), rc);
-		spdk_put_io_channel(hello_context->bdev_io_channel);
-		spdk_bdev_close(hello_context->bdev_desc);
-		spdk_app_stop(-1);
-	}
-}
-
 /*
  * Our initial event that kicks off everything from main().
  */
@@ -294,12 +250,6 @@ hello_start(void *arg1)
 	}
 	snprintf(hello_context->buff, blk_size, "%s", "Hello World!\n");
 
-	if (spdk_bdev_is_zoned(hello_context->bdev)) {
-		hello_reset_zone(hello_context);
-		/* If bdev is zoned, the callback, reset_zone_complete, will call hello_write() */
-		return;
-	}
-
 	hello_write(hello_context);
 }
 
@@ -311,7 +261,7 @@ main(int argc, char **argv)
 	struct hello_context_t hello_context = {};
 
 	/* Set default values in opts structure. */
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "hello_bdev";
 
 	/*

examples/blob/cli/Makefile

@@ -38,6 +38,9 @@ APP = blobcli
 C_SRCS := blobcli.c
 
 # Don't link bdev_lvol in blobcli - otherwise this utility cannot operate on an lvolstore
-SPDK_LIB_LIST = $(filter-out bdev_lvol,$(ALL_MODULES_LIST)) event_bdev
+SPDK_LIB_LIST = $(filter-out bdev_lvol,$(ALL_MODULES_LIST))
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM)
+SPDK_LIB_LIST += bdev accel event thread util conf trace \
+		log jsonrpc json rpc sock notify
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/blob/cli/blobcli.c

@@ -1542,7 +1542,7 @@ main(int argc, char **argv)
 	}
 
 	/* Set default values in opts struct along with name and conf file. */
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "blobcli";
 	opts.json_config_file = cli_context->config_file;
 

examples/blob/hello_world/Makefile

@@ -37,6 +37,9 @@ APP = hello_blob
 
 C_SRCS := hello_blob.c
 
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_bdev
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM)
+SPDK_LIB_LIST += bdev accel event thread util conf trace \
+		log jsonrpc json rpc sock notify
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/blob/hello_world/hello_blob.c

@@ -451,7 +451,7 @@ main(int argc, char **argv)
 	SPDK_NOTICELOG("entry\n");
 
 	/* Set default values in opts structure. */
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 
 	/*
 	 * Setup a few specifics before we init, for most SPDK cmd line

examples/interrupt_tgt/Makefile

@@ -39,10 +39,22 @@ APP = interrupt_tgt
 
 C_SRCS := interrupt_tgt.c
 
-SPDK_LIB_LIST = $(INTR_BLOCKDEV_MODULES_LIST) event_bdev conf
-
-SPDK_LIB_LIST += event_nbd
-SPDK_LIB_LIST += event_vhost
+# Basic libaries to support spdk application
+SPDK_LIB_LIST += event thread
+SPDK_LIB_LIST += jsonrpc json rpc trace conf util log
+# Basic bdev libraries
+SPDK_LIB_LIST += bdev notify accel vmd sock
+SPDK_LIB_LIST += $(EVENT_BDEV_SUBSYSTEM) # event_bdev depends on some other event modules, but they dont support edriven yet
+# Aio bdev library
+SPDK_LIB_LIST += bdev_aio
+# NBD libraries
+SPDK_LIB_LIST += nbd event_nbd
+# some bdev modules don't have pollers, so they can be directly runnning in edriven mode
+SPDK_LIB_LIST += bdev_malloc bdev_passthru bdev_error bdev_gpt bdev_split bdev_raid
+# logical volume and blobstore can directly run in edriven mode
+SPDK_LIB_LIST += bdev_lvol lvol blob_bdev blob
+# blobfs libraries
+SPDK_LIB_LIST += blobfs blobfs_bdev
 
 ifeq ($(SPDK_ROOT_DIR)/lib/env_dpdk,$(CONFIG_ENV))
 SPDK_LIB_LIST += env_dpdk_rpc

examples/interrupt_tgt/interrupt_plugin.py

@@ -1,15 +0,0 @@
-from rpc.client import print_json
-
-
-def reactor_set_interrupt_mode(args):
-    params = {'lcore': args.lcore, 'disable_interrupt': args.disable_interrupt}
-    return args.client.call('reactor_set_interrupt_mode', params)
-
-
-def spdk_rpc_plugin_initialize(subparsers):
-    p = subparsers.add_parser('reactor_set_interrupt_mode',
-                              help="""Set reactor to interrupt or back to poll mode.""")
-    p.add_argument('lcore', type=int, help='lcore of the reactor')
-    p.add_argument('-d', '--disable-interrupt', dest='disable_interrupt', action='store_true',
-                   help='Set reactor back to poll mode')
-    p.set_defaults(func=reactor_set_interrupt_mode)

examples/interrupt_tgt/interrupt_tgt.c

@@ -34,86 +34,17 @@
 #include "spdk/stdinc.h"
 #include "spdk/conf.h"
 #include "spdk/event.h"
-#include "spdk/vhost.h"
-#include "spdk/json.h"
-#include "spdk/jsonrpc.h"
-#include "spdk/rpc.h"
-#include "spdk/env.h"
-
-#include "spdk_internal/event.h"
-
-struct rpc_reactor_set_interrupt_mode {
-	int32_t lcore;
-	bool disable_interrupt;
-};
-
-static const struct spdk_json_object_decoder rpc_reactor_set_interrupt_mode_decoders[] = {
-	{"lcore", offsetof(struct rpc_reactor_set_interrupt_mode, lcore), spdk_json_decode_int32},
-	{"disable_interrupt", offsetof(struct rpc_reactor_set_interrupt_mode, disable_interrupt), spdk_json_decode_bool},
-};
-
-static void
-rpc_reactor_set_interrupt_mode_cb(void *cb_arg)
-{
-	struct spdk_jsonrpc_request *request = cb_arg;
-
-	SPDK_NOTICELOG("complete reactor switch\n");
-
-	spdk_jsonrpc_send_bool_response(request, true);
-}
-
-static void
-rpc_reactor_set_interrupt_mode(struct spdk_jsonrpc_request *request,
-			       const struct spdk_json_val *params)
-{
-	struct rpc_reactor_set_interrupt_mode req = {};
-	int rc;
-
-	if (spdk_json_decode_object(params, rpc_reactor_set_interrupt_mode_decoders,
-				    SPDK_COUNTOF(rpc_reactor_set_interrupt_mode_decoders),
-				    &req)) {
-		SPDK_ERRLOG("spdk_json_decode_object failed\n");
-		spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
-						 "spdk_json_decode_object failed");
-		return;
-	}
-
-	SPDK_NOTICELOG("RPC Start to %s interrupt mode on reactor %d.\n",
-		       req.disable_interrupt ? "disable" : "enable", req.lcore);
-	if (req.lcore >= (int64_t)spdk_env_get_first_core() &&
-	    req.lcore <= (int64_t)spdk_env_get_last_core()) {
-		rc = spdk_reactor_set_interrupt_mode(req.lcore, !req.disable_interrupt,
-						     rpc_reactor_set_interrupt_mode_cb, request);
-		if (rc)	{
-			goto err;
-		}
-	} else {
-		goto err;
-	}
-
-	return;
-
-err:
-	spdk_jsonrpc_send_error_response(request, SPDK_JSONRPC_ERROR_INVALID_PARAMS,
-					 "Invalid parameters");
-}
-/* private */ SPDK_RPC_REGISTER("reactor_set_interrupt_mode", rpc_reactor_set_interrupt_mode,
-				SPDK_RPC_RUNTIME)
 
 static void
 interrupt_tgt_usage(void)
 {
 	printf(" -E                        Set interrupt mode\n");
-	printf(" -S <path>                 directory where to create vhost sockets (default: pwd)\n");
 }
 
 static int
 interrupt_tgt_parse_arg(int ch, char *arg)
 {
 	switch (ch) {
-	case 'S':
-		spdk_vhost_set_socket_path(arg);
-		break;
 	case 'E':
 		spdk_interrupt_mode_enable();
 		break;
@@ -134,10 +65,10 @@ main(int argc, char *argv[])
 	struct spdk_app_opts opts = {};
 	int rc;
 
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "interrupt_tgt";
 
-	if ((rc = spdk_app_parse_args(argc, argv, &opts, "S:E", NULL,
+	if ((rc = spdk_app_parse_args(argc, argv, &opts, "E", NULL,
 				      interrupt_tgt_parse_arg, interrupt_tgt_usage)) !=
 	    SPDK_APP_PARSE_ARGS_SUCCESS) {
 		exit(rc);

examples/ioat/perf/Makefile

@@ -38,6 +38,6 @@ APP = ioat_perf
 
 C_SRCS := perf.c
 
-SPDK_LIB_LIST = ioat util
+SPDK_LIB_LIST = ioat thread util log
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/ioat/perf/perf.c

@@ -520,8 +520,8 @@ int
 main(int argc, char **argv)
 {
 	int rc;
-	struct worker_thread *worker, *main_worker;
-	unsigned main_core;
+	struct worker_thread *worker, *master_worker;
+	unsigned master_core;
 
 	if (parse_args(argc, argv) != 0) {
 		return 1;
@@ -562,22 +562,22 @@ main(int argc, char **argv)
 		goto cleanup;
 	}
 
-	/* Launch all of the secondary workers */
-	main_core = spdk_env_get_current_core();
-	main_worker = NULL;
+	/* Launch all of the slave workers */
+	master_core = spdk_env_get_current_core();
+	master_worker = NULL;
 	worker = g_workers;
 	while (worker != NULL) {
-		if (worker->core != main_core) {
+		if (worker->core != master_core) {
 			spdk_env_thread_launch_pinned(worker->core, work_fn, worker);
 		} else {
-			assert(main_worker == NULL);
-			main_worker = worker;
+			assert(master_worker == NULL);
+			master_worker = worker;
 		}
 		worker = worker->next;
 	}
 
-	assert(main_worker != NULL);
-	rc = work_fn(main_worker);
+	assert(master_worker != NULL);
+	rc = work_fn(master_worker);
 	if (rc != 0) {
 		goto cleanup;
 	}

examples/ioat/verify/Makefile

@@ -38,6 +38,6 @@ APP = verify
 
 C_SRCS := verify.c
 
-SPDK_LIB_LIST = ioat util
+SPDK_LIB_LIST = ioat thread util log
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/ioat/verify/verify.c

@@ -428,8 +428,7 @@ dump_result(struct thread_entry *threads, uint32_t num_threads)
 		total_failed += t->xfer_failed;
 		total_failed += t->fill_failed;
 		if (total_completed || total_failed)
-			printf("lcore = %d, copy success = %" PRIu64 ", copy failed = %" PRIu64 ", fill success = %" PRIu64
-			       ", fill failed = %" PRIu64 "\n",
+			printf("lcore = %d, copy success = %ld, copy failed = %ld, fill success = %ld, fill failed = %ld\n",
 			       t->lcore_id, t->xfer_completed, t->xfer_failed, t->fill_completed, t->fill_failed);
 	}
 	return total_failed ? 1 : 0;

examples/nvme/Makefile

@@ -35,7 +35,7 @@ SPDK_ROOT_DIR := $(abspath $(CURDIR)/../..)
 include $(SPDK_ROOT_DIR)/mk/spdk.common.mk
 
 DIRS-y += hello_world identify perf reconnect nvme_manage arbitration \
-	hotplug cmb_copy abort pmr_persistence
+	hotplug cmb_copy abort
 
 DIRS-$(CONFIG_FIO_PLUGIN) += fio_plugin
 

examples/nvme/abort/abort.c

@@ -106,7 +106,7 @@ static TAILQ_HEAD(, ns_entry) g_namespaces = TAILQ_HEAD_INITIALIZER(g_namespaces
 static int g_num_namespaces;
 static TAILQ_HEAD(, worker_thread) g_workers = TAILQ_HEAD_INITIALIZER(g_workers);
 static int g_num_workers = 0;
-static uint32_t g_main_core;
+static uint32_t g_master_core;
 
 static int g_abort_interval = 1;
 
@@ -479,7 +479,7 @@ work_fn(void *arg)
 			spdk_nvme_qpair_process_completions(ns_ctx->qpair, 0);
 		}
 
-		if (worker->lcore == g_main_core) {
+		if (worker->lcore == g_master_core) {
 			TAILQ_FOREACH(ctrlr_ctx, &worker->ctrlr_ctx, link) {
 				/* Hold mutex to guard ctrlr_ctx->current_queue_depth. */
 				pthread_mutex_lock(&ctrlr_ctx->mutex);
@@ -511,7 +511,7 @@ work_fn(void *arg)
 		}
 	} while (unfinished_ctx > 0);
 
-	if (worker->lcore == g_main_core) {
+	if (worker->lcore == g_master_core) {
 		do {
 			unfinished_ctx = 0;
 
@@ -642,7 +642,7 @@ parse_args(int argc, char **argv)
 	long int val;
 	int rc;
 
-	while ((op = getopt(argc, argv, "a:c:i:o:q:r:s:t:w:GM:T:")) != -1) {
+	while ((op = getopt(argc, argv, "a:c:i:o:q:r:s:t:w:M:")) != -1) {
 		switch (op) {
 		case 'a':
 		case 'i':
@@ -822,17 +822,17 @@ unregister_workers(void)
 
 		TAILQ_FOREACH_SAFE(ns_ctx, &worker->ns_ctx, link, tmp_ns_ctx) {
 			TAILQ_REMOVE(&worker->ns_ctx, ns_ctx, link);
-			printf("NS: %s I/O completed: %" PRIu64 ", failed: %" PRIu64 "\n",
+			printf("NS: %s I/O completed: %lu, failed: %lu\n",
 			       ns_ctx->entry->name, ns_ctx->io_completed, ns_ctx->io_failed);
 			free(ns_ctx);
 		}
 
 		TAILQ_FOREACH_SAFE(ctrlr_ctx, &worker->ctrlr_ctx, link, tmp_ctrlr_ctx) {
 			TAILQ_REMOVE(&worker->ctrlr_ctx, ctrlr_ctx, link);
-			printf("CTRLR: %s abort submitted %" PRIu64 ", failed to submit %" PRIu64 "\n",
+			printf("CTRLR: %s abort submitted %lu, failed to submit %lu\n",
 			       ctrlr_ctx->entry->name, ctrlr_ctx->abort_submitted,
 			       ctrlr_ctx->abort_submit_failed);
-			printf("\t success %" PRIu64 ", unsuccess %" PRIu64 ", failed %" PRIu64 "\n",
+			printf("\t success %lu, unsuccess %lu, failed %lu\n",
 			       ctrlr_ctx->successful_abort, ctrlr_ctx->unsuccessful_abort,
 			       ctrlr_ctx->abort_failed);
 			free(ctrlr_ctx);
@@ -918,14 +918,14 @@ unregister_controllers(void)
 }
 
 static int
-associate_main_worker_with_ctrlr(void)
+associate_master_worker_with_ctrlr(void)
 {
 	struct ctrlr_entry	*entry;
 	struct worker_thread	*worker;
 	struct ctrlr_worker_ctx	*ctrlr_ctx;
 
 	TAILQ_FOREACH(worker, &g_workers, link) {
-		if (worker->lcore == g_main_core) {
+		if (worker->lcore == g_master_core) {
 			break;
 		}
 	}
@@ -957,7 +957,7 @@ get_ctrlr_worker_ctx(struct spdk_nvme_ctrlr *ctrlr)
 	struct ctrlr_worker_ctx *ctrlr_ctx;
 
 	TAILQ_FOREACH(worker, &g_workers, link) {
-		if (worker->lcore == g_main_core) {
+		if (worker->lcore == g_master_core) {
 			break;
 		}
 	}
@@ -1022,7 +1022,7 @@ associate_workers_with_ns(void)
 int main(int argc, char **argv)
 {
 	int rc;
-	struct worker_thread *worker, *main_worker;
+	struct worker_thread *worker, *master_worker;
 	struct spdk_env_opts opts;
 
 	rc = parse_args(argc, argv);
@@ -1070,7 +1070,7 @@ int main(int argc, char **argv)
 		goto cleanup;
 	}
 
-	if (associate_main_worker_with_ctrlr() != 0) {
+	if (associate_master_worker_with_ctrlr() != 0) {
 		rc = -1;
 		goto cleanup;
 	}
@@ -1082,20 +1082,20 @@ int main(int argc, char **argv)
 
 	printf("Initialization complete. Launching workers.\n");
 
-	/* Launch all of the secondary workers */
-	g_main_core = spdk_env_get_current_core();
-	main_worker = NULL;
+	/* Launch all of the slave workers */
+	g_master_core = spdk_env_get_current_core();
+	master_worker = NULL;
 	TAILQ_FOREACH(worker, &g_workers, link) {
-		if (worker->lcore != g_main_core) {
+		if (worker->lcore != g_master_core) {
 			spdk_env_thread_launch_pinned(worker->lcore, work_fn, worker);
 		} else {
-			assert(main_worker == NULL);
-			main_worker = worker;
+			assert(master_worker == NULL);
+			master_worker = worker;
 		}
 	}
 
-	assert(main_worker != NULL);
-	rc = work_fn(main_worker);
+	assert(master_worker != NULL);
+	rc = work_fn(master_worker);
 
 	spdk_env_thread_wait_all();
 

examples/nvme/arbitration/arbitration.c

@@ -1044,8 +1044,8 @@ int
 main(int argc, char **argv)
 {
 	int rc;
-	struct worker_thread *worker, *main_worker;
-	unsigned main_core;
+	struct worker_thread *worker, *master_worker;
+	unsigned master_core;
 	char task_pool_name[30];
 	uint32_t task_count;
 	struct spdk_env_opts opts;
@@ -1099,20 +1099,20 @@ main(int argc, char **argv)
 
 	printf("Initialization complete. Launching workers.\n");
 
-	/* Launch all of the secondary workers */
-	main_core = spdk_env_get_current_core();
-	main_worker = NULL;
+	/* Launch all of the slave workers */
+	master_core = spdk_env_get_current_core();
+	master_worker = NULL;
 	TAILQ_FOREACH(worker, &g_workers, link) {
-		if (worker->lcore != main_core) {
+		if (worker->lcore != master_core) {
 			spdk_env_thread_launch_pinned(worker->lcore, work_fn, worker);
 		} else {
-			assert(main_worker == NULL);
-			main_worker = worker;
+			assert(master_worker == NULL);
+			master_worker = worker;
 		}
 	}
 
-	assert(main_worker != NULL);
-	rc = work_fn(main_worker);
+	assert(master_worker != NULL);
+	rc = work_fn(master_worker);
 
 	spdk_env_thread_wait_all();
 

examples/nvme/fio_plugin/Makefile

@@ -40,6 +40,11 @@ FIO_PLUGIN := spdk_nvme
 
 C_SRCS = fio_plugin.c
 
-SPDK_LIB_LIST = $(SOCK_MODULES_LIST) nvme vmd
+SPDK_LIB_LIST = $(SOCK_MODULES_LIST)
+SPDK_LIB_LIST += nvme thread util log sock vmd jsonrpc json rpc
+
+ifeq ($(CONFIG_RDMA),y)
+SPDK_LIB_LIST += rdma
+endif
 
 include $(SPDK_ROOT_DIR)/mk/spdk.fio.mk

examples/nvme/fio_plugin/README.md

@@ -56,10 +56,8 @@ and instead only use '.'. This is a limitation in fio - it splits filenames on
 ':'. Also, the NVMe namespaces start at 1, not 0, and the namespace must be
 specified at the end of the string.
 
-fio by default forks a separate process for every job. It also supports just spawning a separate
-thread in the same process for every job. The SPDK fio plugin is limited to this latter thread
-usage model, so fio jobs must also specify thread=1 when using the SPDK fio plugin. The SPDK fio
-plugin supports multiple threads - in this case, the "1" just means "use thread mode".
+Currently the SPDK fio plugin is limited to the thread usage model, so fio jobs must also specify thread=1
+when using the SPDK fio plugin.
 
 fio also currently has a race condition on shutdown if dynamically loading the ioengine by specifying the
 engine's full path via the ioengine parameter - LD_PRELOAD is recommended to avoid this race condition.
@@ -139,17 +137,6 @@ then you can reset all zones before fio start running its jobs by using the engi
 
     --initial_zone_reset=1
 
-## Zone Append
-
-When running FIO against a Zoned Namespace you need to specify --iodepth=1 to avoid
-"Zone Invalid Write: The write to a zone was not at the write pointer." I/O errors.
-However, if your controller supports Zone Append, you can use the engine option:
-
-    --zone_append=1
-
-To send zone append commands instead of write commands to the controller.
-When using zone append, you will be able to specify a --iodepth greater than 1.
-
 ## Shared Memory Increase
 
 If your device has a lot of zones, fio can give you errors such as:

examples/nvme/fio_plugin/example_config.fio

@@ -9,6 +9,7 @@ ramp_time=0
 runtime=2
 iodepth=128
 rw=randrw
+bs=4k
 
 [test]
 numjobs=1

examples/nvme/fio_plugin/fio_plugin.c

@@ -93,8 +93,6 @@ struct spdk_fio_options {
 	char	*digest_enable;
 	int	enable_vmd;
 	int	initial_zone_reset;
-	int	zone_append;
-	int	print_qid_mappings;
 };
 
 struct spdk_fio_request {
@@ -132,7 +130,6 @@ struct spdk_fio_qpair {
 	struct spdk_nvme_qpair		*qpair;
 	struct spdk_nvme_ns		*ns;
 	uint32_t			io_flags;
-	bool				zone_append_enabled;
 	bool				nvme_pi_enabled;
 	/* True for DIF and false for DIX, and this is valid only if nvme_pi_enabled is true. */
 	bool				extended_lba;
@@ -290,35 +287,13 @@ pcu(struct spdk_nvme_qpair *qpair, int *completed)
 }
 #endif
 
-static inline uint32_t
-_nvme_get_host_buffer_sector_size(struct spdk_nvme_ns *ns, uint32_t io_flags)
-{
-	bool md_excluded_from_xfer = false;
-	uint32_t md_size;
-	uint32_t ns_flags;
-
-	ns_flags = spdk_nvme_ns_get_flags(ns);
-	md_size = spdk_nvme_ns_get_md_size(ns);
-
-	/* For extended LBA format, if the metadata size is 8 bytes and PRACT is
-	 * enabled(controller inserts/strips PI), we should reduce metadata size
-	 * from block size.
-	 */
-	md_excluded_from_xfer = ((io_flags & SPDK_NVME_IO_FLAGS_PRACT) &&
-				 (ns_flags & SPDK_NVME_NS_EXTENDED_LBA_SUPPORTED) &&
-				 (ns_flags & SPDK_NVME_NS_DPS_PI_SUPPORTED) &&
-				 (md_size == 8));
-
-	return md_excluded_from_xfer ? spdk_nvme_ns_get_sector_size(ns) :
-	       spdk_nvme_ns_get_extended_sector_size(ns);
-}
-
 static void
 attach_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
 	  struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_ctrlr_opts *opts)
 {
 	struct thread_data	*td = cb_ctx;
 	struct spdk_fio_thread	*fio_thread = td->io_ops_data;
+	struct spdk_nvme_io_qpair_opts	qpopts;
 	struct spdk_fio_ctrlr	*fio_ctrlr;
 	struct spdk_fio_qpair	*fio_qpair;
 	struct spdk_nvme_ns	*ns;
@@ -327,7 +302,6 @@ attach_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
 	uint32_t		ns_id;
 	char			*p;
 	long int		tmp;
-	uint32_t		block_size;
 	struct spdk_fio_options *fio_options = td->eo;
 
 	p = strstr(f->file_name, "ns=");
@@ -397,7 +371,20 @@ attach_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
 		return;
 	}
 
-	f->engine_data = fio_qpair;
+	spdk_nvme_ctrlr_get_default_io_qpair_opts(fio_ctrlr->ctrlr, &qpopts, sizeof(qpopts));
+	qpopts.delay_cmd_submit = true;
+	if (fio_options->enable_wrr) {
+		qpopts.qprio = fio_options->wrr_priority;
+	}
+
+	fio_qpair->qpair = spdk_nvme_ctrlr_alloc_io_qpair(fio_ctrlr->ctrlr, &qpopts, sizeof(qpopts));
+	if (!fio_qpair->qpair) {
+		SPDK_ERRLOG("Cannot allocate nvme io_qpair any more\n");
+		g_error = true;
+		free(fio_qpair);
+		return;
+	}
+
 	fio_qpair->ns = ns;
 	fio_qpair->f = f;
 	fio_qpair->fio_ctrlr = fio_ctrlr;
@@ -413,57 +400,6 @@ attach_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
 			fio_qpair->extended_lba ? "extended lba" : "separate metadata");
 	}
 
-	block_size = _nvme_get_host_buffer_sector_size(ns, fio_qpair->io_flags);
-	if (td->o.bs[DDIR_READ] % block_size != 0 || td->o.bs[DDIR_WRITE] % block_size != 0) {
-		if (spdk_nvme_ns_supports_extended_lba(ns)) {
-			SPDK_ERRLOG("--bs has to be a multiple of (LBA data size + Metadata size)\n");
-		} else {
-			SPDK_ERRLOG("--bs has to be a multiple of LBA data size\n");
-		}
-		g_error = true;
-		return;
-	}
-
-	if (fio_options->zone_append && spdk_nvme_ns_get_csi(ns) == SPDK_NVME_CSI_ZNS) {
-		if (spdk_nvme_ctrlr_get_flags(ctrlr) & SPDK_NVME_CTRLR_ZONE_APPEND_SUPPORTED) {
-			fprintf(stdout, "Using zone appends instead of writes on: '%s'\n",
-				fio_qpair->f->file_name);
-			fio_qpair->zone_append_enabled = true;
-		} else {
-			SPDK_WARNLOG("Falling back to writes on: '%s' - ns lacks zone append cmd\n",
-				     fio_qpair->f->file_name);
-		}
-	}
-
-	if (fio_options->initial_zone_reset == 1 && spdk_nvme_ns_get_csi(ns) == SPDK_NVME_CSI_ZNS) {
-#if FIO_HAS_ZBD
-		struct spdk_nvme_qpair *tmp_qpair;
-		int completed = 0, err;
-
-		/* qpair has not been allocated yet (it gets allocated in spdk_fio_open()).
-		 * Create a temporary qpair in order to perform the initial zone reset.
-		 */
-		assert(!fio_qpair->qpair);
-
-		tmp_qpair = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);
-		if (!tmp_qpair) {
-			SPDK_ERRLOG("Cannot allocate a temporary qpair\n");
-			g_error = true;
-			return;
-		}
-
-		err = spdk_nvme_zns_reset_zone(ns, tmp_qpair, 0x0, true, pcu_cb, &completed);
-		if (err || pcu(tmp_qpair, &completed) || completed < 0) {
-			log_err("spdk/nvme: warn: initial_zone_reset: err: %d, cpl: %d\n",
-				err, completed);
-		}
-
-		spdk_nvme_ctrlr_free_io_qpair(tmp_qpair);
-#else
-		log_err("spdk/nvme: ZBD/ZNS is not supported\n");
-#endif
-	}
-
 	f->real_file_size = spdk_nvme_ns_get_size(fio_qpair->ns);
 	if (f->real_file_size <= 0) {
 		g_error = true;
@@ -653,35 +589,39 @@ static int spdk_fio_setup(struct thread_data *td)
 	g_td_count++;
 	pthread_mutex_unlock(&g_mutex);
 
+	if (fio_options->initial_zone_reset == 1) {
+#if FIO_HAS_ZBD
+		struct spdk_fio_qpair *fio_qpair;
+
+		TAILQ_FOREACH(fio_qpair, &fio_thread->fio_qpair, link) {
+			const struct spdk_nvme_zns_ns_data *zns_data;
+			int completed = 0, err;
+
+			if (!fio_qpair->ns) {
+				continue;
+			}
+			zns_data = spdk_nvme_zns_ns_get_data(fio_qpair->ns);
+			if (!zns_data) {
+				continue;
+			}
+
+			err = spdk_nvme_zns_reset_zone(fio_qpair->ns, fio_qpair->qpair, 0x0, true,
+						       pcu_cb, &completed);
+			if (err || pcu(fio_qpair->qpair, &completed) || completed < 0) {
+				log_err("spdk/nvme: warn: initial_zone_reset: err: %d, cpl: %d\n",
+					err, completed);
+			}
+		}
+#else
+		log_err("spdk/nvme: ZBD/ZNS is not supported\n");
+#endif
+	}
+
 	return rc;
 }
 
 static int spdk_fio_open(struct thread_data *td, struct fio_file *f)
 {
-	struct spdk_fio_qpair *fio_qpair = f->engine_data;
-	struct spdk_fio_ctrlr *fio_ctrlr = fio_qpair->fio_ctrlr;
-	struct spdk_fio_options *fio_options = td->eo;
-	struct spdk_nvme_io_qpair_opts	qpopts;
-
-	spdk_nvme_ctrlr_get_default_io_qpair_opts(fio_ctrlr->ctrlr, &qpopts, sizeof(qpopts));
-	qpopts.delay_cmd_submit = true;
-	if (fio_options->enable_wrr) {
-		qpopts.qprio = fio_options->wrr_priority;
-	}
-
-	fio_qpair->qpair = spdk_nvme_ctrlr_alloc_io_qpair(fio_ctrlr->ctrlr, &qpopts, sizeof(qpopts));
-	if (!fio_qpair->qpair) {
-		SPDK_ERRLOG("Cannot allocate nvme io_qpair any more\n");
-		g_error = true;
-		free(fio_qpair);
-		return -1;
-	}
-
-	if (fio_options->print_qid_mappings == 1) {
-		log_info("job %s: %s qid %d\n", td->o.name, f->file_name,
-			 spdk_nvme_qpair_get_id(fio_qpair->qpair));
-	}
-
 	return 0;
 }
 
@@ -740,12 +680,6 @@ static void spdk_fio_io_u_free(struct thread_data *td, struct io_u *io_u)
 	}
 }
 
-static inline uint64_t
-fio_offset_to_zslba(unsigned long long offset, struct spdk_nvme_ns *ns)
-{
-	return (offset / spdk_nvme_zns_ns_get_zone_size(ns)) * spdk_nvme_zns_ns_get_zone_size_sectors(ns);
-}
-
 static int
 fio_extended_lba_setup_pi(struct spdk_fio_qpair *fio_qpair, struct io_u *io_u)
 {
@@ -915,10 +849,6 @@ static void spdk_fio_completion_cb(void *ctx, const struct spdk_nvme_cpl *cpl)
 		}
 	}
 
-	if (spdk_nvme_cpl_is_error(cpl)) {
-		fio_req->io->error = EIO;
-	}
-
 	assert(fio_thread->iocq_count < fio_thread->iocq_size);
 	fio_thread->iocq[fio_thread->iocq_count++] = fio_req->io;
 }
@@ -1003,7 +933,16 @@ spdk_fio_queue(struct thread_data *td, struct io_u *io_u)
 	}
 	fio_req->fio_qpair = fio_qpair;
 
-	block_size = _nvme_get_host_buffer_sector_size(ns, fio_qpair->io_flags);
+	block_size = spdk_nvme_ns_get_extended_sector_size(ns);
+	if ((fio_qpair->io_flags & g_spdk_pract_flag) && (spdk_nvme_ns_get_md_size(ns) == 8)) {
+		/* If metadata size = 8 bytes, PI is stripped (read) or inserted (write), and
+		 *  so reduce metadata size from block size.  (If metadata size > 8 bytes, PI
+		 *  is passed (read) or replaced (write).  So block size is not necessary to
+		 *  change.)
+		 */
+		block_size = spdk_nvme_ns_get_sector_size(ns);
+	}
+
 	lba = io_u->offset / block_size;
 	lba_count = io_u->xfer_buflen / block_size;
 
@@ -1035,31 +974,15 @@ spdk_fio_queue(struct thread_data *td, struct io_u *io_u)
 		break;
 	case DDIR_WRITE:
 		if (!g_spdk_enable_sgl) {
-			if (!fio_qpair->zone_append_enabled) {
-				rc = spdk_nvme_ns_cmd_write_with_md(ns, fio_qpair->qpair, io_u->buf, md_buf, lba,
-								    lba_count,
-								    spdk_fio_completion_cb, fio_req,
-								    fio_qpair->io_flags, dif_ctx->apptag_mask, dif_ctx->app_tag);
-			} else {
-				uint64_t zslba = fio_offset_to_zslba(io_u->offset, fio_qpair->ns);
-				rc = spdk_nvme_zns_zone_append_with_md(ns, fio_qpair->qpair, io_u->buf, md_buf, zslba,
-								       lba_count,
-								       spdk_fio_completion_cb, fio_req,
-								       fio_qpair->io_flags, dif_ctx->apptag_mask, dif_ctx->app_tag);
-			}
+			rc = spdk_nvme_ns_cmd_write_with_md(ns, fio_qpair->qpair, io_u->buf, md_buf, lba,
+							    lba_count,
+							    spdk_fio_completion_cb, fio_req,
+							    fio_qpair->io_flags, dif_ctx->apptag_mask, dif_ctx->app_tag);
 		} else {
-			if (!fio_qpair->zone_append_enabled) {
-				rc = spdk_nvme_ns_cmd_writev_with_md(ns, fio_qpair->qpair, lba,
-								     lba_count, spdk_fio_completion_cb, fio_req, fio_qpair->io_flags,
-								     spdk_nvme_io_reset_sgl, spdk_nvme_io_next_sge, md_buf,
-								     dif_ctx->apptag_mask, dif_ctx->app_tag);
-			} else {
-				uint64_t zslba = fio_offset_to_zslba(io_u->offset, fio_qpair->ns);
-				rc = spdk_nvme_zns_zone_appendv_with_md(ns, fio_qpair->qpair, zslba,
-									lba_count, spdk_fio_completion_cb, fio_req, fio_qpair->io_flags,
-									spdk_nvme_io_reset_sgl, spdk_nvme_io_next_sge, md_buf,
-									dif_ctx->apptag_mask, dif_ctx->app_tag);
-			}
+			rc = spdk_nvme_ns_cmd_writev_with_md(ns, fio_qpair->qpair, lba,
+							     lba_count, spdk_fio_completion_cb, fio_req, fio_qpair->io_flags,
+							     spdk_nvme_io_reset_sgl, spdk_nvme_io_next_sge, md_buf,
+							     dif_ctx->apptag_mask, dif_ctx->app_tag);
 		}
 		break;
 	default:
@@ -1219,6 +1142,18 @@ spdk_fio_get_zoned_model(struct thread_data *td, struct fio_file *f, enum zbd_zo
 	return -EINVAL;
 }
 
+static uint64_t
+spdk_fio_qpair_mdts_nbytes(struct spdk_fio_qpair *fio_qpair)
+{
+	const struct spdk_nvme_ctrlr_data *cdata;
+	union spdk_nvme_cap_register cap;
+
+	cap = spdk_nvme_ctrlr_get_regs_cap(fio_qpair->fio_ctrlr->ctrlr);
+	cdata = spdk_nvme_ctrlr_get_data(fio_qpair->fio_ctrlr->ctrlr);
+
+	return (uint64_t)1 << (12 + cap.bits.mpsmin + cdata->mdts);
+}
+
 static int
 spdk_fio_report_zones(struct thread_data *td, struct fio_file *f, uint64_t offset,
 		      struct zbd_zone *zbdz, unsigned int nr_zones)
@@ -1227,9 +1162,8 @@ spdk_fio_report_zones(struct thread_data *td, struct fio_file *f, uint64_t offse
 	struct spdk_fio_qpair *fio_qpair = NULL;
 	const struct spdk_nvme_zns_ns_data *zns = NULL;
 	struct spdk_nvme_zns_zone_report *report;
-	struct spdk_nvme_qpair *tmp_qpair;
-	uint32_t report_nzones = 0, report_nzones_max, report_nbytes, mdts_nbytes;
-	uint64_t zsze_nbytes, ns_nzones, lba_nbytes;
+	uint32_t report_nzones = 0, report_nzones_max, report_nbytes;
+	uint64_t mdts_nbytes, zsze_nbytes, ns_nzones, lba_nbytes;
 	int completed = 0, err;
 
 	fio_qpair = get_fio_qpair(fio_thread, f);
@@ -1243,19 +1177,8 @@ spdk_fio_report_zones(struct thread_data *td, struct fio_file *f, uint64_t offse
 		return -EINVAL;
 	}
 
-	/* qpair has not been allocated yet (it gets allocated in spdk_fio_open()).
-	 * Create a temporary qpair in order to perform report zones.
-	 */
-	assert(!fio_qpair->qpair);
-
-	tmp_qpair = spdk_nvme_ctrlr_alloc_io_qpair(fio_qpair->fio_ctrlr->ctrlr, NULL, 0);
-	if (!tmp_qpair) {
-		log_err("spdk/nvme: cannot allocate a temporary qpair\n");
-		return -EIO;
-	}
-
 	/** Retrieve device parameters */
-	mdts_nbytes = spdk_nvme_ns_get_max_io_xfer_size(fio_qpair->ns);
+	mdts_nbytes = spdk_fio_qpair_mdts_nbytes(fio_qpair);
 	lba_nbytes = spdk_nvme_ns_get_sector_size(fio_qpair->ns);
 	zsze_nbytes = spdk_nvme_zns_ns_get_zone_size(fio_qpair->ns);
 	ns_nzones = spdk_nvme_zns_ns_get_num_zones(fio_qpair->ns);
@@ -1264,17 +1187,16 @@ spdk_fio_report_zones(struct thread_data *td, struct fio_file *f, uint64_t offse
 	report_nzones_max = (mdts_nbytes - sizeof(*report)) / sizeof(report->descs[0]);
 	report_nzones_max = spdk_min(spdk_min(report_nzones_max, nr_zones), ns_nzones);
 	report_nbytes = sizeof(report->descs[0]) * report_nzones_max + sizeof(*report);
-	report = calloc(1, report_nbytes);
+	report = spdk_dma_zmalloc(report_nbytes, NVME_IO_ALIGN, NULL);
 	if (!report) {
 		log_err("spdk/nvme: failed report_zones(): ENOMEM\n");
-		err = -ENOMEM;
-		goto exit;
+		return -ENOMEM;
 	}
 
-	err = spdk_nvme_zns_report_zones(fio_qpair->ns, tmp_qpair, report, report_nbytes,
+	err = spdk_nvme_zns_report_zones(fio_qpair->ns, fio_qpair->qpair, report, report_nbytes,
 					 offset / lba_nbytes, SPDK_NVME_ZRA_LIST_ALL, true, pcu_cb,
 					 &completed);
-	if (err || pcu(tmp_qpair, &completed) || completed < 0) {
+	if (err || pcu(fio_qpair->qpair, &completed) || completed < 0) {
 		log_err("spdk/nvme: report_zones(): err: %d, cpl: %d\n", err, completed);
 		err = err ? err : -EIO;
 		goto exit;
@@ -1332,8 +1254,7 @@ spdk_fio_report_zones(struct thread_data *td, struct fio_file *f, uint64_t offse
 	}
 
 exit:
-	spdk_nvme_ctrlr_free_io_qpair(tmp_qpair);
-	free(report);
+	spdk_dma_free(report);
 
 	return err ? err : (int)report_nzones;
 }
@@ -1628,26 +1549,6 @@ static struct fio_option options[] = {
 		.category	= FIO_OPT_C_ENGINE,
 		.group		= FIO_OPT_G_INVALID,
 	},
-	{
-		.name		= "zone_append",
-		.lname		= "Use zone append instead of write",
-		.type		= FIO_OPT_INT,
-		.off1		= offsetof(struct spdk_fio_options, zone_append),
-		.def		= "0",
-		.help		= "Use zone append instead of write (zone_append=1 or zone_append=0)",
-		.category	= FIO_OPT_C_ENGINE,
-		.group		= FIO_OPT_G_INVALID,
-	},
-	{
-		.name		= "print_qid_mappings",
-		.lname		= "Print job-to-qid mappings",
-		.type		= FIO_OPT_INT,
-		.off1		= offsetof(struct spdk_fio_options, print_qid_mappings),
-		.def		= "0",
-		.help		= "Print job-to-qid mappings (0=disable, 1=enable)",
-		.category	= FIO_OPT_C_ENGINE,
-		.group		= FIO_OPT_G_INVALID,
-	},
 	{
 		.name		= NULL,
 	},

examples/nvme/identify/identify.c

@@ -1,8 +1,8 @@
 /*-
  *   BSD LICENSE
  *
- *   Copyright (c) Intel Corporation. All rights reserved.
- *   Copyright (c) 2020 Mellanox Technologies LTD. All rights reserved.
+ *   Copyright (c) Intel Corporation.
+ *   All rights reserved.
  *
  *   Redistribution and use in source and binary forms, with or without
  *   modification, are permitted provided that the following conditions
@@ -50,7 +50,6 @@
 #define MAX_DISCOVERY_LOG_ENTRIES	((uint64_t)1000)
 
 #define NUM_CHUNK_INFO_ENTRIES		8
-#define MAX_OCSSD_PU			128
 #define MAX_ZONE_DESC_ENTRIES		8
 
 static int outstanding_commands;
@@ -86,9 +85,11 @@ static uint64_t g_discovery_page_numrec;
 
 static struct spdk_ocssd_geometry_data geometry_data;
 
-static struct spdk_ocssd_chunk_information_entry *g_ocssd_chunk_info_page;
+static struct spdk_ocssd_chunk_information_entry g_ocssd_chunk_info_page[NUM_CHUNK_INFO_ENTRIES ];
 
-static int64_t g_zone_report_limit = 8;
+static struct spdk_nvme_zns_zone_report *g_zone_report;
+static size_t g_zone_report_size;
+static uint64_t g_nr_zones_requested;
 
 static bool g_hex_dump = false;
 
@@ -98,19 +99,16 @@ static int g_dpdk_mem = 0;
 
 static bool g_dpdk_mem_single_seg = false;
 
-static int g_main_core = 0;
+static int g_master_core = 0;
 
 static char g_core_mask[16] = "0x1";
 
 static struct spdk_nvme_transport_id g_trid;
-static char g_hostnqn[SPDK_NVMF_NQN_MAX_LEN + 1];
 
 static int g_controllers_found = 0;
 
 static bool g_vmd = false;
 
-static bool g_ocssd_verbose = false;
-
 static void
 hex_dump(const void *data, size_t size)
 {
@@ -197,6 +195,15 @@ get_zns_zone_report_completion(void *cb_arg, const struct spdk_nvme_cpl *cpl)
 		printf("get zns zone report failed\n");
 	}
 
+	/*
+	 * Since we requested a partial report, verify that the firmware returned the
+	 * correct number of zones.
+	 */
+	if (g_zone_report->nr_zones != g_nr_zones_requested) {
+		printf("Invalid number of zones returned: %"PRIu64" (expected: %"PRIu64")\n",
+		       g_zone_report->nr_zones, g_nr_zones_requested);
+		exit(1);
+	}
 	outstanding_commands--;
 }
 
@@ -228,10 +235,7 @@ get_features(struct spdk_nvme_ctrlr *ctrlr)
 		SPDK_OCSSD_FEAT_MEDIA_FEEDBACK,
 	};
 
-	/* Submit only one GET FEATURES at a time. There is a known issue #1799
-	 * with Google Cloud Platform NVMe SSDs that do not handle overlapped
-	 * GET FEATURES commands correctly.
-	 */
+	/* Submit several GET FEATURES commands and wait for them to complete */
 	outstanding_commands = 0;
 	for (i = 0; i < SPDK_COUNTOF(features_to_get); i++) {
 		if (!spdk_nvme_ctrlr_is_ocssd_supported(ctrlr) &&
@@ -243,12 +247,11 @@ get_features(struct spdk_nvme_ctrlr *ctrlr)
 		} else {
 			printf("get_feature(0x%02X) failed to submit command\n", features_to_get[i]);
 		}
-
-		while (outstanding_commands) {
-			spdk_nvme_ctrlr_process_admin_completions(ctrlr);
-		}
 	}
 
+	while (outstanding_commands) {
+		spdk_nvme_ctrlr_process_admin_completions(ctrlr);
+	}
 }
 
 static int
@@ -549,34 +552,15 @@ get_ocssd_chunk_info_log_page(struct spdk_nvme_ns *ns)
 {
 	struct spdk_nvme_ctrlr *ctrlr = spdk_nvme_ns_get_ctrlr(ns);
 	int nsid = spdk_nvme_ns_get_id(ns);
-	uint32_t num_entry = geometry_data.num_grp * geometry_data.num_pu * geometry_data.num_chk;
-	uint32_t xfer_size = spdk_nvme_ns_get_max_io_xfer_size(ns);
-	uint32_t buf_size = 0;
-	uint64_t buf_offset = 0;
 	outstanding_commands = 0;
 
-	assert(num_entry != 0);
-	if (!g_ocssd_verbose) {
-		num_entry = spdk_min(num_entry, NUM_CHUNK_INFO_ENTRIES);
-	}
-
-	g_ocssd_chunk_info_page = calloc(num_entry, sizeof(struct spdk_ocssd_chunk_information_entry));
-	assert(g_ocssd_chunk_info_page != NULL);
-
-	buf_size = num_entry * sizeof(struct spdk_ocssd_chunk_information_entry);
-	while (buf_size > 0) {
-		xfer_size = spdk_min(buf_size, xfer_size);
-		if (spdk_nvme_ctrlr_cmd_get_log_page(ctrlr, SPDK_OCSSD_LOG_CHUNK_INFO,
-						     nsid, (void *) g_ocssd_chunk_info_page + buf_offset,
-						     xfer_size, buf_offset, get_log_page_completion, NULL) == 0) {
-			outstanding_commands++;
-		} else {
-			printf("get_ocssd_chunk_info_log_page() failed\n");
-			return -1;
-		}
-
-		buf_size -= xfer_size;
-		buf_offset += xfer_size;
+	if (spdk_nvme_ctrlr_cmd_get_log_page(ctrlr, SPDK_OCSSD_LOG_CHUNK_INFO,
+					     nsid, &g_ocssd_chunk_info_page, sizeof(g_ocssd_chunk_info_page), 0,
+					     get_log_page_completion, NULL) == 0) {
+		outstanding_commands++;
+	} else {
+		printf("get_ocssd_chunk_info_log_page() failed\n");
+		return -1;
 	}
 
 	while (outstanding_commands) {
@@ -606,6 +590,39 @@ get_ocssd_geometry(struct spdk_nvme_ns *ns, struct spdk_ocssd_geometry_data *geo
 	}
 }
 
+static void
+get_zns_zone_report(struct spdk_nvme_ns *ns, struct spdk_nvme_qpair *qpair)
+{
+	g_nr_zones_requested = spdk_nvme_zns_ns_get_num_zones(ns);
+	/*
+	 * Rather than getting the whole zone report, which could contain thousands of zones,
+	 * get maximum MAX_ZONE_DESC_ENTRIES, so that we don't flood stdout.
+	 */
+	g_nr_zones_requested = spdk_min(g_nr_zones_requested, MAX_ZONE_DESC_ENTRIES);
+	outstanding_commands = 0;
+
+	g_zone_report_size = sizeof(struct spdk_nvme_zns_zone_report) +
+			     g_nr_zones_requested * sizeof(struct spdk_nvme_zns_zone_desc);
+	g_zone_report = calloc(1, g_zone_report_size);
+	if (g_zone_report == NULL) {
+		printf("Zone report allocation failed!\n");
+		exit(1);
+	}
+
+	if (spdk_nvme_zns_report_zones(ns, qpair, g_zone_report, g_zone_report_size,
+				       0, SPDK_NVME_ZRA_LIST_ALL, true,
+				       get_zns_zone_report_completion, NULL)) {
+		printf("spdk_nvme_zns_report_zones() failed\n");
+		exit(1);
+	} else {
+		outstanding_commands++;
+	}
+
+	while (outstanding_commands) {
+		spdk_nvme_qpair_process_completions(qpair, 0);
+	}
+}
+
 static void
 print_hex_be(const void *v, size_t size)
 {
@@ -650,7 +667,7 @@ print_uint_var_dec(uint8_t *array, unsigned int len)
 		result += (uint64_t)array[i - 1] << (8 * (i - 1));
 		i--;
 	}
-	printf("%" PRIu64, result);
+	printf("%lu", result);
 }
 
 /* Print ASCII string as defined by the NVMe spec */
@@ -674,16 +691,6 @@ print_ascii_string(const void *buf, size_t size)
 	}
 }
 
-/* Underline a "line" with the given marker, e.g. print_uline("=", printf(...)); */
-static void
-print_uline(char marker, int line_len)
-{
-	for (int i = 1; i < line_len; ++i) {
-		putchar(marker);
-	}
-	putchar('\n');
-}
-
 static void
 print_ocssd_chunk_info(struct spdk_ocssd_chunk_information_entry *chk_info, int chk_num)
 {
@@ -707,48 +714,9 @@ print_ocssd_chunk_info(struct spdk_ocssd_chunk_information_entry *chk_info, int
 		printf("Chunk type (write mode):        %s\n", ct_str);
 		printf("Chunk type (size_deviate):      %s\n", chk_info[i].ct.size_deviate ? "Yes" : "No");
 		printf("Wear-level Index:               %d\n", chk_info[i].wli);
-		printf("Starting LBA:                   %" PRIu64 "\n", chk_info[i].slba);
-		printf("Number of blocks in chunk:      %" PRIu64 "\n", chk_info[i].cnlb);
-		printf("Write Pointer:                  %" PRIu64 "\n", chk_info[i].wp);
-	}
-}
-
-static void
-print_ocssd_chunk_info_verbose(struct spdk_ocssd_chunk_information_entry *chk_info)
-{
-	uint32_t pu, chk, i;
-	uint32_t cnt_free, cnt_closed, cnt_open, cnt_offline;
-	uint32_t max_pu = spdk_min(MAX_OCSSD_PU, (geometry_data.num_grp * geometry_data.num_pu));
-	char cs_str[MAX_OCSSD_PU + 1], cs;
-
-	assert(chk_info != NULL);
-	printf("OCSSD Chunk Info Verbose\n");
-	printf("======================\n");
-
-	printf("%4s %-*s %3s %3s %3s %3s\n", "band", max_pu, "chunk state", "fr", "cl", "op", "of");
-	for (chk = 0; chk < geometry_data.num_chk; chk++) {
-		cnt_free = cnt_closed = cnt_open = cnt_offline = 0;
-		for (pu = 0; pu < max_pu; pu++) {
-			i = (pu * geometry_data.num_chk) + chk;
-			if (chk_info[i].cs.free) {
-				cnt_free++;
-				cs = 'f';
-			} else if (chk_info[i].cs.closed) {
-				cnt_closed++;
-				cs = 'c';
-			} else if (chk_info[i].cs.open) {
-				cnt_open++;
-				cs = 'o';
-			} else if (chk_info[i].cs.offline) {
-				cnt_offline++;
-				cs = 'l';
-			} else {
-				cs = '.';
-			}
-			cs_str[pu] = cs;
-		}
-		cs_str[pu] = 0;
-		printf("%4d %s %3d %3d %3d %3d\n", chk, cs_str, cnt_free, cnt_closed, cnt_open, cnt_offline);
+		printf("Starting LBA:                   %ld\n", chk_info[i].slba);
+		printf("Number of blocks in chunk:      %ld\n", chk_info[i].cnlb);
+		printf("Write Pointer:                  %ld\n", chk_info[i].wp);
 	}
 }
 
@@ -791,71 +759,20 @@ print_ocssd_geometry(struct spdk_ocssd_geometry_data *geometry_data)
 }
 
 static void
-print_zns_zone(struct spdk_nvme_zns_zone_desc *desc)
+print_zns_zone_report(void)
 {
-	printf("ZSLBA: 0x%016"PRIx64" ZCAP: 0x%016"PRIx64" WP: 0x%016"PRIx64" ZS: %x ZT: %x ZA: %x\n",
-	       desc->zslba, desc->zcap, desc->wp, desc->zs, desc->zt, desc->za.raw);
-}
+	uint64_t i;
 
-static void
-get_and_print_zns_zone_report(struct spdk_nvme_ns *ns, struct spdk_nvme_qpair *qpair)
-{
-	struct spdk_nvme_zns_zone_report *report_buf;
-	size_t report_bufsize;
-	uint64_t zone_size_lba = spdk_nvme_zns_ns_get_zone_size_sectors(ns);
-	uint64_t total_zones = spdk_nvme_zns_ns_get_num_zones(ns);
-	uint64_t max_zones_per_buf, zones_to_print, i;
-	uint64_t handled_zones = 0;
-	uint64_t slba = 0;
+	printf("NVMe ZNS Zone Report Glance\n");
+	printf("===========================\n");
 
-	outstanding_commands = 0;
-
-	report_bufsize = spdk_nvme_ns_get_max_io_xfer_size(ns);
-	report_buf = calloc(1, report_bufsize);
-	if (!report_buf) {
-		printf("Zone report allocation failed!\n");
-		exit(1);
+	for (i = 0; i < g_zone_report->nr_zones; i++) {
+		struct spdk_nvme_zns_zone_desc *desc = &g_zone_report->descs[i];
+		printf("Zone: %"PRIu64" ZSLBA: 0x%016"PRIx64" ZCAP: 0x%016"PRIx64" WP: 0x%016"PRIx64" ZS: %x ZT: %x ZA: %x\n",
+		       i, desc->zslba, desc->zcap, desc->wp, desc->zs, desc->zt, desc->za.raw);
 	}
-
-	zones_to_print = g_zone_report_limit ? spdk_min(total_zones, (uint64_t)g_zone_report_limit) : \
-			 total_zones;
-
-	print_uline('=', printf("NVMe ZNS Zone Report (first %zu of %zu)\n", zones_to_print, total_zones));
-
-	while (handled_zones < zones_to_print) {
-		memset(report_buf, 0, report_bufsize);
-
-		if (spdk_nvme_zns_report_zones(ns, qpair, report_buf, report_bufsize,
-					       slba, SPDK_NVME_ZRA_LIST_ALL, true,
-					       get_zns_zone_report_completion, NULL)) {
-			fprintf(stderr, "spdk_nvme_zns_report_zones() failed\n");
-			exit(1);
-		} else {
-			outstanding_commands++;
-		}
-
-		while (outstanding_commands) {
-			spdk_nvme_qpair_process_completions(qpair, 0);
-		}
-
-		max_zones_per_buf = (report_bufsize - sizeof(*report_buf)) / sizeof(report_buf->descs[0]);
-		if (report_buf->nr_zones > max_zones_per_buf) {
-			fprintf(stderr, "nr_zones too big\n");
-			exit(1);
-		}
-
-		if (!report_buf->nr_zones) {
-			break;
-		}
-
-		for (i = 0; i < report_buf->nr_zones && handled_zones < zones_to_print; i++) {
-			print_zns_zone(&report_buf->descs[i]);
-			slba += zone_size_lba;
-			handled_zones++;
-		}
-	}
-
-	free(report_buf);
+	free(g_zone_report);
+	g_zone_report = NULL;
 }
 
 static void
@@ -896,24 +813,6 @@ print_zns_ns_data(const struct spdk_nvme_zns_ns_data *nsdata_zns)
 	printf("\n");
 }
 
-static const char *
-csi_name(enum spdk_nvme_csi csi)
-{
-	switch (csi) {
-	case SPDK_NVME_CSI_NVM:
-		return "NVM";
-	case SPDK_NVME_CSI_KV:
-		return "KV";
-	case SPDK_NVME_CSI_ZNS:
-		return "ZNS";
-	default:
-		if (csi >= 0x30 && csi <= 0x3f) {
-			return "Vendor specific";
-		}
-		return "Unknown";
-	}
-}
-
 static void
 print_namespace(struct spdk_nvme_ctrlr *ctrlr, struct spdk_nvme_ns *ns)
 {
@@ -925,7 +824,6 @@ print_namespace(struct spdk_nvme_ctrlr *ctrlr, struct spdk_nvme_ns *ns)
 	uint32_t				flags;
 	char					uuid_str[SPDK_UUID_STRING_LEN];
 	uint32_t				blocksize;
-	enum spdk_nvme_dealloc_logical_block_read_value	dlfeat_read_value;
 
 	cdata = spdk_nvme_ctrlr_get_data(ctrlr);
 	nsdata = spdk_nvme_ns_get_data(ns);
@@ -942,16 +840,13 @@ print_namespace(struct spdk_nvme_ctrlr *ctrlr, struct spdk_nvme_ns *ns)
 	/* This function is only called for active namespaces. */
 	assert(spdk_nvme_ns_is_active(ns));
 
-	printf("Command Set Identifier:                %s (%02Xh)\n",
-	       csi_name(spdk_nvme_ns_get_csi(ns)), spdk_nvme_ns_get_csi(ns));
 	printf("Deallocate:                            %s\n",
 	       (flags & SPDK_NVME_NS_DEALLOCATE_SUPPORTED) ? "Supported" : "Not Supported");
 	printf("Deallocated/Unwritten Error:           %s\n",
 	       nsdata->nsfeat.dealloc_or_unwritten_error ? "Supported" : "Not Supported");
-	dlfeat_read_value = spdk_nvme_ns_get_dealloc_logical_block_read_value(ns);
 	printf("Deallocated Read Value:                %s\n",
-	       dlfeat_read_value == SPDK_NVME_DEALLOC_READ_00 ? "All 0x00" :
-	       dlfeat_read_value == SPDK_NVME_DEALLOC_READ_FF ? "All 0xFF" :
+	       nsdata->dlfeat.bits.read_value == SPDK_NVME_DEALLOC_READ_00 ? "All 0x00" :
+	       nsdata->dlfeat.bits.read_value == SPDK_NVME_DEALLOC_READ_FF ? "All 0xFF" :
 	       "Unknown");
 	printf("Deallocate in Write Zeroes:            %s\n",
 	       nsdata->dlfeat.bits.write_zero_deallocate ? "Supported" : "Not Supported");
@@ -1048,11 +943,7 @@ print_namespace(struct spdk_nvme_ctrlr *ctrlr, struct spdk_nvme_ns *ns)
 		get_ocssd_geometry(ns, &geometry_data);
 		print_ocssd_geometry(&geometry_data);
 		get_ocssd_chunk_info_log_page(ns);
-		if (g_ocssd_verbose) {
-			print_ocssd_chunk_info_verbose(g_ocssd_chunk_info_page);
-		} else {
-			print_ocssd_chunk_info(g_ocssd_chunk_info_page, NUM_CHUNK_INFO_ENTRIES);
-		}
+		print_ocssd_chunk_info(g_ocssd_chunk_info_page, NUM_CHUNK_INFO_ENTRIES);
 	} else if (spdk_nvme_ns_get_csi(ns) == SPDK_NVME_CSI_ZNS) {
 		struct spdk_nvme_qpair *qpair = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);
 		if (qpair == NULL) {
@@ -1060,7 +951,8 @@ print_namespace(struct spdk_nvme_ctrlr *ctrlr, struct spdk_nvme_ns *ns)
 			exit(1);
 		}
 		print_zns_ns_data(nsdata_zns);
-		get_and_print_zns_zone_report(ns, qpair);
+		get_zns_zone_report(ns, qpair);
+		print_zns_zone_report();
 		spdk_nvme_ctrlr_free_io_qpair(qpair);
 	}
 }
@@ -1170,7 +1062,6 @@ print_controller(struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_transport
 	union spdk_nvme_cap_register		cap;
 	union spdk_nvme_vs_register		vs;
 	union spdk_nvme_cmbsz_register		cmbsz;
-	union spdk_nvme_pmrcap_register		pmrcap;
 	uint8_t					str[512];
 	uint32_t				i, j;
 	struct spdk_nvme_error_information_entry *error_entry;
@@ -1178,14 +1069,11 @@ print_controller(struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_transport
 	struct spdk_pci_device			*pci_dev;
 	struct spdk_pci_id			pci_id;
 	uint32_t				nsid;
-	uint64_t				pmrsz;
 	struct spdk_nvme_ana_group_descriptor	*desc;
 
 	cap = spdk_nvme_ctrlr_get_regs_cap(ctrlr);
 	vs = spdk_nvme_ctrlr_get_regs_vs(ctrlr);
 	cmbsz = spdk_nvme_ctrlr_get_regs_cmbsz(ctrlr);
-	pmrcap = spdk_nvme_ctrlr_get_regs_pmrcap(ctrlr);
-	pmrsz = spdk_nvme_ctrlr_get_pmrsz(ctrlr);
 
 	if (!spdk_nvme_ctrlr_is_discovery(ctrlr)) {
 		/*
@@ -1298,9 +1186,6 @@ print_controller(struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_transport
 	       (uint64_t)1 << (12 + cap.bits.mpsmin));
 	printf("Memory Page Size Maximum:              %" PRIu64 " bytes\n",
 	       (uint64_t)1 << (12 + cap.bits.mpsmax));
-	printf("Persistent Memory Region:              %s\n",
-	       cap.bits.pmrs ? "Supported" : "Not Supported");
-
 	printf("Optional Asynchronous Events Supported\n");
 	printf("  Namespace Attribute Notices:         %s\n",
 	       cdata->oaes.ns_attribute_notices ? "Supported" : "Not Supported");
@@ -1322,7 +1207,7 @@ print_controller(struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_transport
 		size *= (0x1000 << (cmbsz.bits.szu * 4));
 
 		printf("Supported:                             Yes\n");
-		printf("Total Size:                            %" PRIu64 " bytes\n", size);
+		printf("Total Size:                            %lu bytes\n", size);
 		printf("Submission Queues in CMB:              %s\n",
 		       cmbsz.bits.sqs ? "Supported" : "Not Supported");
 		printf("Completion Queues in CMB:              %s\n",
@@ -1336,20 +1221,6 @@ print_controller(struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_transport
 	}
 	printf("\n");
 
-	printf("Persistent Memory Region Support\n");
-	printf("================================\n");
-	if (cap.bits.pmrs != 0) {
-		printf("Supported:                             Yes\n");
-		printf("Total Size:                            %" PRIu64 " bytes\n", pmrsz);
-		printf("Read data and metadata in PMR          %s\n",
-		       pmrcap.bits.rds ? "Supported" : "Not Supported");
-		printf("Write data and metadata in PMR:        %s\n",
-		       pmrcap.bits.wds ? "Supported" : "Not Supported");
-	} else {
-		printf("Supported:                             No\n");
-	}
-	printf("\n");
-
 	printf("Admin Command Set Attributes\n");
 	printf("============================\n");
 	printf("Security Send/Receive:                 %s\n",
@@ -1904,18 +1775,18 @@ print_controller(struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_transport
 	if (spdk_nvme_ctrlr_is_log_page_supported(ctrlr, SPDK_NVME_INTEL_LOG_TEMPERATURE)) {
 		printf("Intel Temperature Information\n");
 		printf("==================\n");
-		printf("Current Temperature: %" PRIu64 "\n", intel_temperature_page.current_temperature);
-		printf("Overtemp shutdown Flag for last critical component temperature: %" PRIu64 "\n",
+		printf("Current Temperature: %lu\n", intel_temperature_page.current_temperature);
+		printf("Overtemp shutdown Flag for last critical component temperature: %lu\n",
 		       intel_temperature_page.shutdown_flag_last);
-		printf("Overtemp shutdown Flag for life critical component temperature: %" PRIu64 "\n",
+		printf("Overtemp shutdown Flag for life critical component temperature: %lu\n",
 		       intel_temperature_page.shutdown_flag_life);
-		printf("Highest temperature: %" PRIu64 "\n", intel_temperature_page.highest_temperature);
-		printf("Lowest temperature: %" PRIu64 "\n", intel_temperature_page.lowest_temperature);
-		printf("Specified Maximum Operating Temperature: %" PRIu64 "\n",
+		printf("Highest temperature: %lu\n", intel_temperature_page.highest_temperature);
+		printf("Lowest temperature: %lu\n", intel_temperature_page.lowest_temperature);
+		printf("Specified Maximum Operating Temperature: %lu\n",
 		       intel_temperature_page.specified_max_op_temperature);
-		printf("Specified Minimum Operating Temperature: %" PRIu64 "\n",
+		printf("Specified Minimum Operating Temperature: %lu\n",
 		       intel_temperature_page.specified_min_op_temperature);
-		printf("Estimated offset: %" PRId64 "\n", (int64_t)intel_temperature_page.estimated_offset);
+		printf("Estimated offset: %ld\n", intel_temperature_page.estimated_offset);
 		printf("\n");
 		printf("\n");
 
@@ -2040,7 +1911,6 @@ usage(const char *program_name)
 	printf("     traddr      Transport address (e.g. 192.168.100.8)\n");
 	printf("     trsvcid     Transport service identifier (e.g. 4420)\n");
 	printf("     subnqn      Subsystem NQN (default: %s)\n", SPDK_NVMF_DISCOVERY_NQN);
-	printf("     hostnqn     Host NQN\n");
 	printf("    Example: -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.100.8 trsvcid:4420'\n");
 
 	spdk_log_usage(stdout, "-L");
@@ -2050,7 +1920,7 @@ usage(const char *program_name)
 	printf(" -d         DPDK huge memory size in MB\n");
 	printf(" -g         use single file descriptor for DPDK memory segments\n");
 	printf(" -x         print hex dump of raw data\n");
-	printf(" -z         For NVMe Zoned Namespaces, dump the full zone report (-z) or the first N entries (-z N)\n");
+	printf(" -v         verbose (enable warnings)\n");
 	printf(" -V         enumerate VMD\n");
 	printf(" -H         show this usage\n");
 }
@@ -2059,12 +1929,11 @@ static int
 parse_args(int argc, char **argv)
 {
 	int op, rc;
-	char *hostnqn;
 
 	spdk_nvme_trid_populate_transport(&g_trid, SPDK_NVME_TRANSPORT_PCIE);
 	snprintf(g_trid.subnqn, sizeof(g_trid.subnqn), "%s", SPDK_NVMF_DISCOVERY_NQN);
 
-	while ((op = getopt(argc, argv, "d:gi:op:r:xz::HL:V")) != -1) {
+	while ((op = getopt(argc, argv, "d:gi:p:r:xHL:V")) != -1) {
 		switch (op) {
 		case 'd':
 			g_dpdk_mem = spdk_strtol(optarg, 10);
@@ -2083,57 +1952,23 @@ parse_args(int argc, char **argv)
 				return g_shm_id;
 			}
 			break;
-		case 'o':
-			g_ocssd_verbose = true;
-			break;
 		case 'p':
-			g_main_core = spdk_strtol(optarg, 10);
-			if (g_main_core < 0) {
+			g_master_core = spdk_strtol(optarg, 10);
+			if (g_master_core < 0) {
 				fprintf(stderr, "Invalid core number\n");
-				return g_main_core;
+				return g_master_core;
 			}
-			snprintf(g_core_mask, sizeof(g_core_mask), "0x%llx", 1ULL << g_main_core);
+			snprintf(g_core_mask, sizeof(g_core_mask), "0x%llx", 1ULL << g_master_core);
 			break;
 		case 'r':
 			if (spdk_nvme_transport_id_parse(&g_trid, optarg) != 0) {
 				fprintf(stderr, "Error parsing transport address\n");
 				return 1;
 			}
-
-			assert(optarg != NULL);
-			hostnqn = strcasestr(optarg, "hostnqn:");
-			if (hostnqn) {
-				size_t len;
-
-				hostnqn += strlen("hostnqn:");
-
-				len = strcspn(hostnqn, " \t\n");
-				if (len > (sizeof(g_hostnqn) - 1)) {
-					fprintf(stderr, "Host NQN is too long\n");
-					return 1;
-				}
-
-				memcpy(g_hostnqn, hostnqn, len);
-				g_hostnqn[len] = '\0';
-			}
 			break;
 		case 'x':
 			g_hex_dump = true;
 			break;
-		case 'z':
-			if (optarg == NULL && argv[optind] != NULL && argv[optind][0] != '-') {
-				g_zone_report_limit = spdk_strtol(argv[optind], 10);
-				++optind;
-			} else if (optarg) {
-				g_zone_report_limit = spdk_strtol(optarg, 10);
-			} else {
-				g_zone_report_limit = 0;
-			}
-			if (g_zone_report_limit < 0) {
-				fprintf(stderr, "Invalid Zone Report limit\n");
-				return g_zone_report_limit;
-			}
-			break;
 		case 'L':
 			rc = spdk_log_set_flag(optarg);
 			if (rc < 0) {
@@ -2147,7 +1982,7 @@ parse_args(int argc, char **argv)
 			break;
 		case 'H':
 			usage(argv[0]);
-			exit(EXIT_SUCCESS);
+			break;
 		case 'V':
 			g_vmd = true;
 			break;
@@ -2164,7 +1999,6 @@ static bool
 probe_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
 	 struct spdk_nvme_ctrlr_opts *opts)
 {
-	memcpy(opts->hostnqn, g_hostnqn, sizeof(opts->hostnqn));
 	return true;
 }
 
@@ -2193,7 +2027,7 @@ int main(int argc, char **argv)
 	opts.shm_id = g_shm_id;
 	opts.mem_size = g_dpdk_mem;
 	opts.mem_channel = 1;
-	opts.main_core = g_main_core;
+	opts.master_core = g_master_core;
 	opts.core_mask = g_core_mask;
 	opts.hugepage_single_segments = g_dpdk_mem_single_seg;
 	if (g_trid.trtype != SPDK_NVME_TRANSPORT_PCIE) {
@@ -2211,11 +2045,7 @@ int main(int argc, char **argv)
 
 	/* A specific trid is required. */
 	if (strlen(g_trid.traddr) != 0) {
-		struct spdk_nvme_ctrlr_opts opts;
-
-		spdk_nvme_ctrlr_get_default_ctrlr_opts(&opts, sizeof(opts));
-		memcpy(opts.hostnqn, g_hostnqn, sizeof(opts.hostnqn));
-		ctrlr = spdk_nvme_connect(&g_trid, &opts, sizeof(opts));
+		ctrlr = spdk_nvme_connect(&g_trid, NULL, 0);
 		if (!ctrlr) {
 			fprintf(stderr, "spdk_nvme_connect() failed\n");
 			return 1;

examples/nvme/nvme_manage/nvme_manage.c

@@ -909,7 +909,7 @@ opal_dump_info(struct spdk_opal_d0_features_info *feat)
 		printf("Opal Geometry feature:\n");
 		printf("Align = %s", (feat->geo.alignment_granularity ? "Y, " : "N, "));
 		printf("Logical block size = %d, ", from_be32(&feat->geo.logical_block_size));
-		printf("Lowest aligned LBA = %" PRIu64 "\n", from_be64(&feat->geo.lowest_aligned_lba));
+		printf("Lowest aligned LBA = %ld\n", from_be64(&feat->geo.lowest_aligned_lba));
 		printf("\n");
 	}
 
@@ -1104,8 +1104,8 @@ opal_setup_lockingrange(struct dev *iter)
 			info = spdk_opal_get_locking_range_info(iter->opal_dev, locking_range_id);
 
 			printf("\nlocking range ID: %d\n", info->locking_range_id);
-			printf("range start: %" PRIu64 "\n", info->range_start);
-			printf("range length: %" PRIu64 "\n", info->range_length);
+			printf("range start: %ld\n", info->range_start);
+			printf("range length: %ld\n", info->range_length);
 			printf("read lock enabled: %d\n", info->read_lock_enabled);
 			printf("write lock enabled: %d\n", info->write_lock_enabled);
 			printf("read locked: %d\n", info->read_locked);
@@ -1168,8 +1168,8 @@ opal_list_locking_ranges(struct dev *iter)
 				printf("locking range ID: %d\t", info->locking_range_id);
 				if (i == 0) { printf("(Global Range)"); }
 				printf("\n===============================================\n");
-				printf("range start: %" PRIu64 "\t", info->range_start);
-				printf("range length: %" PRIu64 "\n", info->range_length);
+				printf("range start: %ld\t", info->range_start);
+				printf("range length: %ld\n", info->range_length);
 				printf("read lock enabled: %d\t", info->read_lock_enabled);
 				printf("write lock enabled: %d\t", info->write_lock_enabled);
 				printf("read locked: %d\t", info->read_locked);

examples/nvme/pmr_persistence/Makefile

@@ -1,38 +0,0 @@
-#
-#  BSD LICENSE
-#
-#  Copyright (c) Samsung Electronics Co., Ltd.
-#  All rights reserved.
-#
-#  Redistribution and use in source and binary forms, with or without
-#  modification, are permitted provided that the following conditions
-#  are met:
-#
-#    * Redistributions of source code must retain the above copyright
-#      notice, this list of conditions and the following disclaimer.
-#    * Redistributions in binary form must reproduce the above copyright
-#      notice, this list of conditions and the following disclaimer in
-#      the documentation and/or other materials provided with the
-#      distribution.
-#    * Neither the name of Samsung Electronics Co., Ltd. nor the names of its
-#      contributors may be used to endorse or promote products derived
-#      from this software without specific prior written permission.
-#
-#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-#  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-#  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-#  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-#  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-#  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-#  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-#  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-#  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-#  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-#  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-#
-
-SPDK_ROOT_DIR := $(abspath $(CURDIR)/../../..)
-
-APP = pmr_persistence
-
-include $(SPDK_ROOT_DIR)/mk/nvme.libtest.mk

examples/nvme/pmr_persistence/pmr_persistence.c

@@ -1,419 +0,0 @@
-/*-
- *   BSD LICENSE
- *
- *   Copyright (c) Samsung Electronics Co., Ltd.
- *   All rights reserved.
- *
- *   Redistribution and use in source and binary forms, with or without
- *   modification, are permitted provided that the following conditions
- *   are met:
- *
- *     * Redistributions of source code must retain the above copyright
- *       notice, this list of conditions and the following disclaimer.
- *     * Redistributions in binary form must reproduce the above copyright
- *       notice, this list of conditions and the following disclaimer in
- *       the documentation and/or other materials provided with the
- *       distribution.
- *     * Neither the name of Samsung Electronics Co., Ltd.,  nor the names of its
- *       contributors may be used to endorse or promote products derived
- *       from this software without specific prior written permission.
- *
- *   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- *   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- *   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- *   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- *   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- *   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- *   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- *   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- *   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- *   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- *   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-
-#include "spdk/stdinc.h"
-
-#include "spdk/env.h"
-#include "spdk/nvme.h"
-#include "spdk/string.h"
-
-struct nvme_io {
-	struct spdk_nvme_ctrlr *ctrlr;
-	struct spdk_nvme_transport_id trid;
-	struct spdk_nvme_ns *ns;
-	unsigned nsid;
-	unsigned rlba;
-	unsigned nlbas;
-	unsigned wlba;
-	uint32_t lba_size;
-	unsigned done;
-};
-
-struct config {
-	struct nvme_io pmr_dev;
-	size_t         copy_size;
-};
-
-static struct config g_config;
-
-/* Namespaces index from 1. Return 0 to invoke an error */
-static unsigned
-get_nsid(const struct spdk_nvme_transport_id *trid)
-{
-	if (!strcmp(trid->traddr, g_config.pmr_dev.trid.traddr)) {
-		return g_config.pmr_dev.nsid;
-	}
-	return 0;
-}
-
-static void
-check_io(void *arg, const struct spdk_nvme_cpl *completion)
-{
-	g_config.pmr_dev.done = 1;
-}
-
-static int
-pmr_persistence(void)
-{
-	int rc = 0;
-	void *pmr_buf, *buf;
-	size_t sz;
-	struct spdk_nvme_qpair	*qpair;
-
-	/* Allocate Queue Pair for the Controller with PMR */
-	qpair = spdk_nvme_ctrlr_alloc_io_qpair(g_config.pmr_dev.ctrlr, NULL, 0);
-	if (qpair == NULL) {
-		printf("ERROR: spdk_nvme_ctrlr_alloc_io_qpair() failed\n");
-		return -ENOMEM;
-	}
-
-	/* Enable the PMR */
-	rc = spdk_nvme_ctrlr_enable_pmr(g_config.pmr_dev.ctrlr);
-	if (rc) {
-		printf("ERROR: Enabling PMR failed\n");
-		printf("Are you sure %s has a valid PMR?\n",
-		       g_config.pmr_dev.trid.traddr);
-		goto free_qpair;
-	}
-
-	/* Allocate buffer from PMR */
-	pmr_buf = spdk_nvme_ctrlr_map_pmr(g_config.pmr_dev.ctrlr, &sz);
-	if (pmr_buf == NULL || sz < g_config.copy_size) {
-		printf("ERROR: PMR buffer allocation failed\n");
-		rc = -ENOMEM;
-		goto disable_pmr;
-	}
-
-	/* Clear the done flag */
-	g_config.pmr_dev.done = 0;
-
-	/* Do the write to the PMR IO buffer, reading from rlba */
-	rc = spdk_nvme_ns_cmd_read(g_config.pmr_dev.ns, qpair, pmr_buf,
-				   g_config.pmr_dev.rlba, g_config.pmr_dev.nlbas,
-				   check_io, NULL, 0);
-	if (rc != 0) {
-		fprintf(stderr, "Read I/O to PMR failed\n");
-		rc = -EIO;
-		goto unmap_pmr;
-	}
-	while (!g_config.pmr_dev.done) {
-		spdk_nvme_qpair_process_completions(qpair, 0);
-	}
-
-	/* Clear the done flag */
-	g_config.pmr_dev.done = 0;
-
-	pmr_buf = NULL;
-
-	/* Free PMR buffer */
-	rc = spdk_nvme_ctrlr_unmap_pmr(g_config.pmr_dev.ctrlr);
-	if (rc) {
-		printf("ERROR: Unmapping PMR failed\n");
-		goto disable_pmr;
-	}
-
-	/* Disable the PMR */
-	rc = spdk_nvme_ctrlr_disable_pmr(g_config.pmr_dev.ctrlr);
-	if (rc) {
-		printf("ERROR: Disabling PMR failed\n");
-		goto free_qpair;
-	}
-
-	/* Free the queue */
-	spdk_nvme_ctrlr_free_io_qpair(qpair);
-
-	rc = spdk_nvme_ctrlr_reset(g_config.pmr_dev.ctrlr);
-	if (rc) {
-		printf("ERROR: Resetting Controller failed\n");
-		return rc;
-	}
-
-	/* Allocate Queue Pair for the Controller with PMR */
-	qpair = spdk_nvme_ctrlr_alloc_io_qpair(g_config.pmr_dev.ctrlr, NULL, 0);
-	if (qpair == NULL) {
-		printf("ERROR: spdk_nvme_ctrlr_alloc_io_qpair() failed\n");
-		return -ENOMEM;
-	}
-
-	/* Enable the PMR */
-	rc = spdk_nvme_ctrlr_enable_pmr(g_config.pmr_dev.ctrlr);
-	if (rc) {
-		printf("ERROR: Enabling PMR failed\n");
-		goto free_qpair;
-	}
-
-	/* Allocate buffer from PMR */
-	pmr_buf = spdk_nvme_ctrlr_map_pmr(g_config.pmr_dev.ctrlr, &sz);
-	if (pmr_buf == NULL || sz < g_config.copy_size) {
-		printf("ERROR: PMR buffer allocation failed\n");
-		rc = -ENOMEM;
-		goto disable_pmr;
-	}
-
-	/* Do the read from the PMR IO buffer, write to wlba */
-	rc = spdk_nvme_ns_cmd_write(g_config.pmr_dev.ns, qpair, pmr_buf,
-				    g_config.pmr_dev.wlba, g_config.pmr_dev.nlbas,
-				    check_io, NULL, 0);
-	if (rc != 0) {
-		fprintf(stderr, "Read I/O from PMR failed\n");
-		rc = -EIO;
-		goto unmap_pmr;
-	}
-	while (!g_config.pmr_dev.done) {
-		spdk_nvme_qpair_process_completions(qpair, 0);
-	}
-
-	/* Clear the done flag */
-	g_config.pmr_dev.done = 0;
-
-	buf = spdk_zmalloc(g_config.copy_size, 0x1000, NULL, SPDK_ENV_SOCKET_ID_ANY, SPDK_MALLOC_DMA);
-	if (buf == NULL) {
-		printf("ERROR: Buffer allocation failed\n");
-		rc = -ENOMEM;
-		goto unmap_pmr;
-	}
-
-	/* Do the read from wlba to a buffer */
-	rc = spdk_nvme_ns_cmd_read(g_config.pmr_dev.ns, qpair, buf,
-				   g_config.pmr_dev.wlba, g_config.pmr_dev.nlbas,
-				   check_io, NULL, 0);
-	if (rc != 0) {
-		fprintf(stderr, "Read I/O from WLBA failed\n");
-		rc = -EIO;
-		goto free_buf;
-	}
-	while (!g_config.pmr_dev.done) {
-		spdk_nvme_qpair_process_completions(qpair, 0);
-	}
-
-	/* Clear the done flag */
-	g_config.pmr_dev.done = 0;
-
-	/* Compare the data in the read buffer to the PMR buffer */
-	if (memcmp(buf, pmr_buf, g_config.copy_size)) {
-		printf("PMR Data Not Persistent, after Controller Reset\n");
-		rc = -EIO;
-	} else {
-		printf("PMR Data is Persistent across Controller Reset\n");
-	}
-
-free_buf:
-	spdk_free(buf);
-
-unmap_pmr:
-	/* Free PMR buffer */
-	spdk_nvme_ctrlr_unmap_pmr(g_config.pmr_dev.ctrlr);
-
-disable_pmr:
-	/* Disable the PMR */
-	spdk_nvme_ctrlr_disable_pmr(g_config.pmr_dev.ctrlr);
-
-free_qpair:
-	/* Free the queue */
-	spdk_nvme_ctrlr_free_io_qpair(qpair);
-
-	return rc;
-}
-
-static bool
-probe_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
-	 struct spdk_nvme_ctrlr_opts *opts)
-{
-	/* We will only attach to the Controller specified by the user */
-	if (spdk_nvme_transport_id_compare(trid, &g_config.pmr_dev.trid)) {
-		printf("%s - not probed %s!\n", __func__, trid->traddr);
-		return 0;
-	}
-
-	printf("%s - probed %s!\n", __func__, trid->traddr);
-	return 1;
-}
-
-static void
-attach_cb(void *cb_ctx, const struct spdk_nvme_transport_id *trid,
-	  struct spdk_nvme_ctrlr *ctrlr, const struct spdk_nvme_ctrlr_opts *opts)
-{
-	struct spdk_nvme_ns *ns;
-
-	ns = spdk_nvme_ctrlr_get_ns(ctrlr, get_nsid(trid));
-	if (ns == NULL) {
-		fprintf(stderr, "Could not locate namespace %d on controller %s.\n",
-			get_nsid(trid), trid->traddr);
-		exit(-1);
-	}
-
-	g_config.pmr_dev.ctrlr    = ctrlr;
-	g_config.pmr_dev.ns       = ns;
-	g_config.pmr_dev.lba_size = spdk_nvme_ns_get_sector_size(ns);
-
-	printf("%s - attached %s!\n", __func__, trid->traddr);
-}
-
-static void
-usage(char *program_name)
-{
-	printf("%s options (all mandatory)", program_name);
-	printf("\n");
-	printf("\t[-p PCIe address of the NVMe Device with PMR support]\n");
-	printf("\t[-n Namespace ID]\n");
-	printf("\t[-r Read LBA]\n");
-	printf("\t[-l Number of LBAs to read]\n");
-	printf("\t[-w Write LBA]\n");
-	printf("\n");
-}
-
-static int
-parse_args(int argc, char **argv)
-{
-	int op;
-	unsigned num_args = 0;
-	long int val;
-
-	while ((op = getopt(argc, argv, "p:n:r:l:w:")) != -1) {
-		switch (op) {
-		case 'p':
-			snprintf(&g_config.pmr_dev.trid.traddr[0], SPDK_NVMF_TRADDR_MAX_LEN + 1,
-				 "%s", optarg);
-
-			g_config.pmr_dev.trid.trtype = SPDK_NVME_TRANSPORT_PCIE;
-
-			spdk_nvme_transport_id_populate_trstring(&g_config.pmr_dev.trid,
-					spdk_nvme_transport_id_trtype_str(g_config.pmr_dev.trid.trtype));
-
-			num_args++;
-			break;
-		case 'n':
-		case 'r':
-		case 'l':
-		case 'w':
-			val = spdk_strtol(optarg, 10);
-			if (val < 0) {
-				fprintf(stderr, "Converting a string to integer failed\n");
-				return val;
-			}
-			switch (op) {
-			case 'n':
-				g_config.pmr_dev.nsid = (unsigned)val;
-				num_args++;
-				break;
-			case 'r':
-				g_config.pmr_dev.rlba = (unsigned)val;
-				num_args++;
-				break;
-			case 'l':
-				g_config.pmr_dev.nlbas = (unsigned)val;
-				num_args++;
-				break;
-			case 'w':
-				g_config.pmr_dev.wlba = (unsigned)val;
-				num_args++;
-				break;
-			}
-			break;
-		default:
-			usage(argv[0]);
-			return 1;
-		}
-	}
-
-	if (num_args != 5) {
-		usage(argv[0]);
-		return 1;
-	}
-
-	return 0;
-}
-
-static void
-cleanup(void)
-{
-	struct spdk_nvme_detach_ctx *detach_ctx = NULL;
-
-	spdk_nvme_detach_async(g_config.pmr_dev.ctrlr, &detach_ctx);
-
-	while (detach_ctx && spdk_nvme_detach_poll_async(detach_ctx) == -EAGAIN) {
-		;
-	}
-}
-
-int main(int argc, char **argv)
-{
-	int rc = 0;
-	struct spdk_env_opts opts;
-
-	/*
-	 * Parse the input arguments. For now we use the following
-	 * format list:
-	 *
-	 * -p <pci id> -n <namespace> -r <Read LBA> -l <number of LBAs> -w <Write LBA>
-	 *
-	 */
-	rc = parse_args(argc, argv);
-	if (rc) {
-		fprintf(stderr, "Error in parse_args(): %d\n", rc);
-		return rc;
-	}
-
-	/*
-	 * SPDK relies on an abstraction around the local environment
-	 * named env that handles memory allocation and PCI device operations.
-	 * This library must be initialized first.
-	 *
-	 */
-	spdk_env_opts_init(&opts);
-	opts.name = "pmr_persistence";
-	opts.shm_id = 0;
-	if (spdk_env_init(&opts) < 0) {
-		fprintf(stderr, "Unable to initialize SPDK env\n");
-		return 1;
-	}
-
-	/*
-	 * PMRs only apply to PCIe attached NVMe controllers so we
-	 * only probe the PCIe bus. This is the default when we pass
-	 * in NULL for the first argument.
-	 */
-
-	rc = spdk_nvme_probe(NULL, NULL, probe_cb, attach_cb, NULL);
-	if (rc) {
-		fprintf(stderr, "Error in spdk_nvme_probe(): %d\n", rc);
-		cleanup();
-		return rc;
-	}
-
-	g_config.copy_size = g_config.pmr_dev.nlbas * g_config.pmr_dev.lba_size;
-
-	/*
-	 * Call the pmr_persistence() function which performs the data copy
-	 * to PMR region, resets the Controller and verifies the data persistence
-	 * or returns an error code if it fails.
-	 */
-	rc = pmr_persistence();
-	if (rc) {
-		fprintf(stderr, "Error in pmr_persistence(): %d\n", rc);
-	}
-
-	cleanup();
-
-	return rc;
-}

examples/nvme/reconnect/reconnect.c

@@ -115,7 +115,6 @@ static bool g_warn;
 static uint32_t g_keep_alive_timeout_in_ms = 0;
 static uint8_t g_transport_retry_count = 4;
 static uint8_t g_transport_ack_timeout = 0; /* disabled */
-static bool g_dpdk_mem_single_seg = false;
 
 static const char *g_core_mask;
 
@@ -266,9 +265,6 @@ build_nvme_name(char *name, size_t length, struct spdk_nvme_ctrlr *ctrlr)
 	case SPDK_NVME_TRANSPORT_TCP:
 		snprintf(name, length, "TCP (addr:%s subnqn:%s)", trid->traddr, trid->subnqn);
 		break;
-	case SPDK_NVME_TRANSPORT_VFIOUSER:
-		snprintf(name, length, "VFIOUSER (%s)", trid->traddr);
-		break;
 	case SPDK_NVME_TRANSPORT_CUSTOM:
 		snprintf(name, length, "CUSTOM (%s)", trid->traddr);
 		break;
@@ -688,7 +684,7 @@ parse_args(int argc, char **argv)
 	g_core_mask = NULL;
 	g_max_completions = 0;
 
-	while ((op = getopt(argc, argv, "c:gm:o:q:r:k:s:t:w:A:GM:R:T:")) != -1) {
+	while ((op = getopt(argc, argv, "c:m:o:q:r:k:s:t:w:A:GM:R:T:")) != -1) {
 		switch (op) {
 		case 'm':
 		case 'o':
@@ -738,9 +734,6 @@ parse_args(int argc, char **argv)
 		case 'c':
 			g_core_mask = optarg;
 			break;
-		case 'g':
-			g_dpdk_mem_single_seg = true;
-			break;
 		case 'r':
 			if (add_trid(optarg)) {
 				usage(argv[0]);
@@ -1073,8 +1066,8 @@ nvme_poll_ctrlrs(void *arg)
 int main(int argc, char **argv)
 {
 	int rc;
-	struct worker_thread *worker, *main_worker;
-	unsigned main_core;
+	struct worker_thread *worker, *master_worker;
+	unsigned master_core;
 	struct spdk_env_opts opts;
 	pthread_t thread_id = 0;
 
@@ -1092,7 +1085,6 @@ int main(int argc, char **argv)
 	if (g_dpdk_mem) {
 		opts.mem_size = g_dpdk_mem;
 	}
-	opts.hugepage_single_segments = g_dpdk_mem_single_seg;
 	if (spdk_env_init(&opts) < 0) {
 		fprintf(stderr, "Unable to initialize SPDK env\n");
 		rc = 1;
@@ -1133,20 +1125,20 @@ int main(int argc, char **argv)
 
 	printf("Initialization complete. Launching workers.\n");
 
-	/* Launch all of the secondary workers */
-	main_core = spdk_env_get_current_core();
-	main_worker = NULL;
+	/* Launch all of the slave workers */
+	master_core = spdk_env_get_current_core();
+	master_worker = NULL;
 	TAILQ_FOREACH(worker, &g_workers, link) {
-		if (worker->lcore != main_core) {
+		if (worker->lcore != master_core) {
 			spdk_env_thread_launch_pinned(worker->lcore, work_fn, worker);
 		} else {
-			assert(main_worker == NULL);
-			main_worker = worker;
+			assert(master_worker == NULL);
+			master_worker = worker;
 		}
 	}
 
-	assert(main_worker != NULL);
-	rc = work_fn(main_worker);
+	assert(master_worker != NULL);
+	rc = work_fn(master_worker);
 
 	spdk_env_thread_wait_all();
 

examples/nvmf/nvmf/Makefile

@@ -38,6 +38,15 @@ include $(SPDK_ROOT_DIR)/mk/spdk.modules.mk
 APP := nvmf
 
 C_SRCS := nvmf.c
-SPDK_LIB_LIST = $(ALL_MODULES_LIST) event_bdev nvmf
+SPDK_LIB_LIST = $(ALL_MODULES_LIST)
+SPDK_LIB_LIST += nvmf thread util bdev conf accel rpc jsonrpc json log sock trace notify
+SPDK_LIB_LIST += event $(EVENT_BDEV_SUBSYSTEM)
+
+ifeq ($(CONFIG_FC),y)
+ifneq ($(strip $(CONFIG_FC_PATH)),)
+SYS_LIBS += -L$(CONFIG_FC_PATH)
+endif
+SYS_LIBS += -lufc
+endif
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/nvmf/nvmf/nvmf.c

@@ -90,7 +90,7 @@ TAILQ_HEAD(, nvmf_reactor) g_reactors = TAILQ_HEAD_INITIALIZER(g_reactors);
 TAILQ_HEAD(, nvmf_target_poll_group) g_poll_groups = TAILQ_HEAD_INITIALIZER(g_poll_groups);
 static uint32_t g_num_poll_groups = 0;
 
-static struct nvmf_reactor *g_main_reactor = NULL;
+static struct nvmf_reactor *g_master_reactor = NULL;
 static struct nvmf_reactor *g_next_reactor = NULL;
 static struct spdk_thread *g_init_thread = NULL;
 static struct spdk_thread *g_fini_thread = NULL;
@@ -183,8 +183,6 @@ parse_args(int argc, char **argv, struct spdk_env_opts *opts)
 			opts->no_pci = true;
 			break;
 		case 'h':
-			usage(argv[0]);
-			exit(EXIT_SUCCESS);
 		default:
 			usage(argv[0]);
 			return 1;
@@ -334,7 +332,7 @@ nvmf_init_threads(void)
 	char thread_name[32];
 	struct nvmf_reactor *nvmf_reactor;
 	struct spdk_cpuset cpumask;
-	uint32_t main_core = spdk_env_get_current_core();
+	uint32_t master_core = spdk_env_get_current_core();
 
 	/* Whenever SPDK creates a new lightweight thread it will call
 	 * nvmf_schedule_spdk_thread asking for the application to begin
@@ -373,9 +371,9 @@ nvmf_init_threads(void)
 
 		TAILQ_INSERT_TAIL(&g_reactors, nvmf_reactor, link);
 
-		if (i == main_core) {
-			g_main_reactor = nvmf_reactor;
-			g_next_reactor = g_main_reactor;
+		if (i == master_core) {
+			g_master_reactor = nvmf_reactor;
+			g_next_reactor = g_master_reactor;
 		} else {
 			rc = spdk_env_thread_launch_pinned(i,
 							   nvmf_reactor_run,
@@ -389,8 +387,8 @@ nvmf_init_threads(void)
 
 	/* Spawn a lightweight thread only on the current core to manage this application. */
 	spdk_cpuset_zero(&cpumask);
-	spdk_cpuset_set_cpu(&cpumask, main_core, true);
-	snprintf(thread_name, sizeof(thread_name), "nvmf_main_thread");
+	spdk_cpuset_set_cpu(&cpumask, master_core, true);
+	snprintf(thread_name, sizeof(thread_name), "nvmf_master_thread");
 	g_init_thread = spdk_thread_create(thread_name, &cpumask);
 	if (!g_init_thread) {
 		fprintf(stderr, "failed to create spdk thread\n");
@@ -711,13 +709,7 @@ static void
 nvmf_subsystem_init_done(int rc, void *cb_arg)
 {
 	fprintf(stdout, "bdev subsystem init successfully\n");
-
-	rc = spdk_rpc_initialize(g_rpc_addr);
-	if (rc) {
-		spdk_app_stop(rc);
-		return;
-	}
-
+	spdk_rpc_initialize(g_rpc_addr);
 	spdk_rpc_set_state(SPDK_RPC_RUNTIME);
 
 	g_target_state = NVMF_INIT_TARGET;
@@ -895,7 +887,7 @@ int main(int argc, char **argv)
 	rc = nvmf_init_threads();
 	assert(rc == 0);
 
-	/* Send a message to the thread assigned to the main reactor
+	/* Send a message to the thread assigned to the master reactor
 	 * that continues initialization. This is how we bootstrap the
 	 * program so that all code from here on is running on an SPDK thread.
 	 */
@@ -906,7 +898,7 @@ int main(int argc, char **argv)
 
 	spdk_thread_send_msg(g_init_thread, nvmf_target_app_start, NULL);
 
-	nvmf_reactor_run(g_main_reactor);
+	nvmf_reactor_run(g_master_reactor);
 
 	spdk_env_thread_wait_all();
 	nvmf_destroy_threads();

examples/sock/hello_world/Makefile

@@ -38,6 +38,6 @@ APP = hello_sock
 C_SRCS := hello_sock.c
 
 SPDK_LIB_LIST = $(SOCK_MODULES_LIST)
-SPDK_LIB_LIST += event_net sock
+SPDK_LIB_LIST += event_net net event thread util conf trace log jsonrpc json rpc sock notify
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk

examples/sock/hello_world/hello_sock.c

@@ -88,7 +88,7 @@ hello_sock_usage(void)
 	printf(" -P port       port number\n");
 	printf(" -N sock_impl  socket implementation, e.g., -N posix or -N uring\n");
 	printf(" -S            start in server mode\n");
-	printf(" -V            print out additional informations\n");
+	printf(" -V            print out additional informations");
 }
 
 /*
@@ -410,7 +410,7 @@ main(int argc, char **argv)
 	struct hello_context_t hello_context = {};
 
 	/* Set default values in opts structure. */
-	spdk_app_opts_init(&opts, sizeof(opts));
+	spdk_app_opts_init(&opts);
 	opts.name = "hello_sock";
 	opts.shutdown_cb = hello_sock_shutdown_cb;
 

examples/vmd/led/Makefile

@@ -39,6 +39,6 @@ APP = led
 
 C_SRCS := led.c
 
-SPDK_LIB_LIST = vmd log util
+SPDK_LIB_LIST = vmd log
 
 include $(SPDK_ROOT_DIR)/mk/spdk.app.mk