2017-02-15 23:45:58 +00:00
# Changelog
2016-03-08 20:34:24 +00:00
2019-04-30 15:25:32 +00:00
## v19.07: (Upcoming Release)
2019-07-26 12:20:58 +00:00
### ftl
EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache
to allow for recovery from dirty shutdown event.
Added handling of Asynchronous Nand Management Events (ANM).
2019-07-26 10:55:49 +00:00
### vmd
EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated
controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected
NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into
SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect
NVMe driver to the device located at the given transport ID.
To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used.
Current implementation does not support hotplug.
2019-06-11 00:59:22 +00:00
### blobfs
Blobfs file asynchronous operations were added to public APIs.
2019-05-07 06:35:01 +00:00
### util
A new file API `spdk_posix_file_load` was added to load file content into a data buffer.
2019-07-18 00:26:21 +00:00
New APIs `spdk_dif_ctx_set_data_offset` , `spdk_dif_verify_stream` ,
`spdk_dif_get_range_with_md` , `spdk_dif_get_length_with_md` have been added,
and existing APIs `spdk_dif_ctx_init` , `spdk_dif_set_md_interleave_iovs` , and
`spdk_dif_generate_stream` have been refined to insert or strip DIF by stream
fasion with any alignment.
New APIs `spdk_dif_ctx_set_remapped_init_ref_tag` , `spdk_dif_remap_ref_tag` ,
and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag.
New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been
added to compute CRC-32C checksum for extended LBA payload.
2019-07-29 12:45:14 +00:00
Bdevperf and bdevio applications now support starting tests with application specific
RPCs. Please see helper Python scripts in their respective directories.
This is a move towards simpler RPC-only configuration for all main
and auxiliary applications.
Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10,
and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or
RPC driven run-time configuration.
2019-07-29 20:06:52 +00:00
### nvmf
2019-04-22 09:28:02 +00:00
2019-07-29 20:06:52 +00:00
EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been
added. This depends on the FC HBA driver at
https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation ](https://spdk.io/doc/nvmf.html#nvmf_fc_transport )
for more information.
2019-04-26 21:25:20 +00:00
2019-05-23 07:24:46 +00:00
Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation
state is stored in a JSON file on the local filesystem between target restart. To support this,
an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added.
This allows the user to specify which file to store the persistent reservation state in. Note
that this is done per namespace.
2019-06-11 15:07:28 +00:00
The c2h success optimization under which a command capsule response is not sent
2019-07-15 02:58:55 +00:00
for reads is turned on by default. A config knob was added to allow disabling
the optimization. This will mostly be used for integration testing with 5.0.x kernels
while some compatibility fixes make their way down the pipeline for 5.1.x kernels.
2019-06-11 15:07:28 +00:00
2019-07-15 12:58:16 +00:00
The sock priority setting of the TCP connection owned by the tcp transport is added. It is
used to optimize the TCP connection performance under designated traffic classes. And the
priority is used to differeniate the sock priority between SPDK NVMe-oF TCP target application
and other TCP based applications.
2019-04-26 21:25:20 +00:00
Shared receive queue can now be disabled even for NICs that support it using the
`nvmf_create_transport` RPC method parameter `no_srq` . The actual use of a shared
receive queue is predicated on hardware support when this flag is not used.
2019-05-15 12:40:12 +00:00
spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal
poll group for the qpair. And `ConnectionScheduler` configuration is added into the
[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection
scheduling strategy among different spdk threads.
2019-04-15 09:54:38 +00:00
Added infrastructure to retrieve global and per poll group NVMf statistics.
2019-07-18 00:26:21 +00:00
DIF strip and insert is now supported for TCP transport. When it is enabled, DIF
setting is not exposed to the NVMe-oF initiator, and DIF is attached into data
for write I/O and stripped from data for read I/O.
Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and
updated the related rpc function nvmf_create_transport to make this
configurable parameter available to users. The `dif_insert_or_strip` is relevant
for TCP transport for now and used to configure the DIF strip and insert.
2019-05-23 09:04:49 +00:00
Added infrastructure to retrieve NVMf transport statistics.
2019-05-05 10:11:40 +00:00
### notify
2019-05-05 10:20:43 +00:00
The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were
renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()` ,
respectively. And update type name of callback accordingly.
2019-05-05 10:11:40 +00:00
2019-04-29 15:02:40 +00:00
### bdev
The format of the data returned by the get_bdevs_iostat RPC has changed to
make it easier to parse. It now returns an object with a "ticks" object
and "bdevs" array with the per-bdev statistics.
2019-05-07 22:43:11 +00:00
A new bdev module `delay` has been added which simulates a drive latency when placed
on top of a Null bdev. This module is intended only for testing and can be created using
the new RPC `bdev_delay_create` . That RPC takes the name of the underlying bdev as well
as average and p99 latency arguments for both read and write operations. Average latency is
defined as a value close to what you would expect a perf tool such as FIO to report back as
the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one
would expect the drive to see the slowest 1% of I/O report. For underlying drives with already
significant latency, the latency values provided to the drive will be additive. This should be
taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device.
2019-07-18 00:26:21 +00:00
DIF reference tag remapping is now supported for partition type virtual bdev
modules. When using partition type virtual bdevs, block address space is
remapped during I/O processing and DIF reference tag is remapped accordingly.
2019-07-25 08:28:10 +00:00
Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in
separate buffer. To check support for separatate metadata, use spdk_bdev_is_md_separate().
All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID,
2019-07-23 23:01:34 +00:00
one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically
generated may change.
A new virtual bdev module `compress` has been added to provide compression services on top of
a thinly provisioned logical volume. See documentation for complete details.
2019-07-25 08:28:10 +00:00
2019-05-24 22:07:32 +00:00
### nvme
Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a
previously attached controller.
2019-05-14 15:00:11 +00:00
Nvme Opal library spdk_opal_cmd deprecated. Adding seperate command APIs.
2019-05-10 15:24:07 +00:00
NVMe Opal library add support for activating locking SP which will make the transaction
from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking
of this method, lock and unlock features will be enabled.
2019-06-11 14:57:09 +00:00
NVMe Opal library add support for locking/unlocking range and list locking range info.
2019-07-09 09:48:09 +00:00
NVMe opal library add support for multiuser. Admin can enable user and add user to specific
locking range and the user can lock/unlock his range.
2019-05-14 15:00:11 +00:00
2019-06-12 17:22:01 +00:00
Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass
a completely formed command to an NVMe submission queue (buffer addresses and all).
This is supported on the PCIe transport only.
Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address
of the NVMe controller registers. This is supported on the PCIe transport only.
Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter
structure to allow caller to override the virtual and optionally physical address
of the submission and completion queue pair to be created. This is supported on
the PCIe transport only.
2019-07-25 08:28:10 +00:00
Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables
logging of failed requests. By default logging is enabled.
Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and
spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages.
Added support for NVMe Sanitize command.
2019-06-04 04:22:00 +00:00
### env
The parameter `free_space` has been added to spdk_ring_enqueue() to wait when
the ring is almost full and resume when there is enough space available in
the ring.
2019-06-28 08:18:30 +00:00
A new API `spdk_mempool_lookup` has been added to lookup the memory pool created
by the primary process.
2019-07-29 09:58:42 +00:00
Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow
iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs
to attach/detach PCI devices are no longer thread safe. They are now meant to
be called from only a single thread only, the same only that called spdk_env_init().
This applies to the newly added APIs as well.
2019-07-29 09:37:38 +00:00
### vpp
SPDK now supports VPP version 19.04.2, up from VPP 18.01.
VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library).
This allows for better control over sessions and queues.
Please see VPP documentation for more details:
[VPP Host Stack ](https://wiki.fd.io/view/VPP/HostStack )
2019-05-14 12:50:15 +00:00
### sock
Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for
this socket. When a socket is created, it is often assigned to a sock group using
spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently.
For some network devices, it is optimal to assign particular sockets to specific
sock groups. This API is intended to provide the user with that information.
2019-05-14 18:40:20 +00:00
spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group.
spdk_sock_group_create() is updated to allow input the user provided ctx.
2019-07-15 12:04:45 +00:00
spdk_sock_set_priority() is added to set the priority of the socket.
2019-05-14 18:40:20 +00:00
2019-03-02 08:32:19 +00:00
### rpc
Added thread_get_stats RPC method to retrieve existing statistics.
2019-04-15 09:54:38 +00:00
Added nvmf_get_stats RPC method to retrieve NVMf susbsystem statistics.
2019-07-29 09:58:42 +00:00
Response buffers for RPC requests are now always pre-allocated, which implies
that all spdk_jsonrpc_begin_result() calls always succeed and return a valid
buffer for JSON response. RPC calls no longer need to check if the buffer is
non-NULL.
Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when
renaming existing RPC. First time a deprecated alias is used, it will print
a warning message.
RPC `get_rpc_methods` was renamed `rpc_get_methods` . The old name is still usable,
but is now deprecated.
2019-06-18 12:34:36 +00:00
### blobstore
A snapshot can now be deleted if there is only a single clone on top of it.
2019-07-29 09:58:42 +00:00
### build
2019-07-29 20:25:55 +00:00
Preliminary support for cross compilation is now available. Targeting an older
CPU on the same architecture using your native compiler can be accomplished by
using the `--target-arch` option to `configure` as follows:
2019-07-29 09:58:42 +00:00
2019-07-29 20:25:55 +00:00
~~~
./configure --target-arch=broadwell
~~~
Additionally, some support for cross-compiling to other architectures has been
added via the `--cross-prefix` argument to `configure` . To cross-compile, set CC
and CXX to the cross compilers, then run configure as follows:
~~~
./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu
~~~
2019-07-29 09:58:42 +00:00
### vhost
By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork
located inside SPDK repo. The internal fork is still kept around to support older
DPDK versions, but is considered legacy and will be eventually removed.
`configure` will now automatically use the upstream rte_vhost if the used DPDK
version is >= 19.05.
spdk_vhost_init() is now asynchronous and accepts a completion callback.
2019-07-25 08:28:10 +00:00
### thread
Exposed spdk_set_thread() in order for applications to associate
with SPDK thread when necessary.
Added spdk_thread_destroy() to allow framework polling the thread to
release resources associated with that thread.
2019-04-30 15:22:46 +00:00
## v19.04:
2019-01-29 09:22:50 +00:00
2019-01-30 02:16:32 +00:00
### nvme
2019-04-30 16:48:03 +00:00
Added asynchronous probe support. New APIs spdk_nvme_probe_async(),
spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable
this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a
context associated with the specified controllers. Users then call
2019-03-05 04:29:37 +00:00
spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation
2019-04-30 16:48:03 +00:00
completed.
2019-01-30 02:16:32 +00:00
2019-03-06 20:23:56 +00:00
A new qpair creation option, delay_pcie_doorbell, was added. This can be passed
2019-04-30 16:48:03 +00:00
to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as
spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the
doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions().
This can result in significantly fewer MMIO writes to the doorbell register
under heavy load, greatly improving performance.
2019-03-06 20:23:56 +00:00
2019-04-30 16:48:03 +00:00
spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller
memory buffer, if available.
2019-02-25 17:15:52 +00:00
2019-04-30 16:48:03 +00:00
spdk_nvme_ctrlr_get_flags() was added to return controller feature
flags. Two flags are currently tracked:
SPDK_NVME_CTRLR_SGL_SUPPORTED
SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED
The NVMe hotplug poller is now able to detach devices hot-removed from the system
2019-03-23 21:38:14 +00:00
via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind` .
2019-04-30 16:48:03 +00:00
Opal support was added for scan, take ownership, revert TPer, and dumping device
info. The nvme_manage tool can be used to perform these operations. The public
API functions are spdk_nvme_ctrlr_security_receive() and
spdk_nvme_ctrlr_security_send(). This module should be considered experimental
pending additional features and tests.
2019-07-18 01:00:37 +00:00
The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB
if the corresponding NVMe-oF target allows.
2018-12-19 18:04:18 +00:00
### raid
Added new strip_size_kb rpc param on create to replace the more ambiguous
strip_size. The strip_size rpc param is deprecated.
2019-04-26 22:30:49 +00:00
Changed the raid bdev product_name from "Pooled Device" to "Raid Volume"
2019-03-01 14:13:38 +00:00
### thread
2019-04-30 16:52:23 +00:00
Added spdk_thread_has_pollers() function to verify if there are any registered
pollers to be run on the thread. Added spdk_thread_is_idle() function to check
if there are any scheduled operations to be performed on the thread at given
time.
spdk_thread_create() now takes an optional CPU affinity mask that is passed to
the scheduler.
spdk_thread_lib_init() now takes an optional context size. For each thread
created, an additional region of memory of this size will be allocated. A
pointer to this region of memory can be obtained by calling
spdk_thread_get_ctx(). The inverse operation is also available via
spdk_thread_get_from_ctx().
spdk_thread_poll() now optionally accepts the current time, in ticks. This can
be used to avoid some calls to spdk_get_ticks() internally.
2019-02-22 13:58:46 +00:00
2019-04-30 14:05:32 +00:00
spdk_app_start() now only accepts a single context argument.
2019-03-03 22:28:23 +00:00
### bdev
An new API `spdk_bdev_get_data_block_size` has been added to get size of data
block except for metadata.
2019-04-03 12:59:05 +00:00
spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used
instead.
2017-11-07 22:05:19 +00:00
A mechanism for acquiring and releasing data buffers from bdev modules, used
to perform zero copy operations, was added.
2019-04-22 08:43:58 +00:00
New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(),
spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been
added to get metadata and DIF settings.
2019-04-30 15:10:56 +00:00
Bdevs claimed by the `examine_config` callback will be now further examined in the
`examine_disk` callback.
2019-04-30 15:14:22 +00:00
spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel
from a bdev_io.
2019-07-29 09:51:37 +00:00
### ocf
Updated OCF submodule to OCF v19.3.2
Added support for many-to-one configuration for OCF bdev.
Multiple core devices can now be cached on single cache device.
Added persistent metadata support, allowing to restore cache state after shutdown.
During start of SPDK application, the devices are examined and if OCF metadata
is present - appropriate OCF bdevs will be recreated.
Added Write-Back mode support. In this mode, data is first written to
caching device and periodically synchronized to the core devices.
Dirty data is saved as persistent metadata on cache device,
allowing for safe restore during application restart.
For more details please see OCF documentation:
[OpenCAS cache configuration ](https://open-cas.github.io/cache_configuration.html )
nvmf/rdma: Add shared receive queue support
This is a new feature for NVMEoF RDMA target, that is intended to save
resource allocation (by sharing them) and utilize the
locality (completions and memory) to get the best performance with
Shared Receive Queues (SRQs). We'll create a SRQ per core (poll
group), per device and associate each created QP/CQ with an
appropriate SRQ.
Our testing environment has 2 hosts.
Host 1:
CPU: Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz dual socket (8 cores total)
Network: ConnectX-5, ConnectX-5 VPI , 100GbE, single-port QSFP28, PCIe3.0 x16
Disk: Intel Optane SSD 900P Series
OS: Fedora 27 x86_64
Host 2:
CPU: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz dual-socket (24 cores total)
Network: ConnectX-4 VPI , 100GbE, dual-port QSFP28
Disk: Intel Optane SSD 900P Series
OS : CentOS 7.5.1804 x86_64
Hosts are connected via Spectrum switch.
Host 1 is running SPDK NVMeoF target.
Host 2 is used as initiator running fio with SPDK plugin.
Configuration:
- SPDK NVMeoF target: cpu mask 0x0F (4 cores), max queue depth 128,
max SRQ depth 1024, max QPs per controller 1024
- Single NVMf subsystem with single namespace backed by physical SSD disk
- fio with SPDK plugin: randread pattern, 1-256 jobs, block size 4k,
IO depth 16, cpu_mask 0xFFF0, IO rate 10k, rate process “poisson”
Here is a full fio command line:
fio --name=Job --stats=1 --group_reporting=1 --idle-prof=percpu \
--loops=1 --numjobs=1 --thread=1 --time_based=1 --runtime=30s \
--ramp_time=5s --bs=4k --size=4G --iodepth=16 --readwrite=randread \
--rwmixread=75 --randrepeat=1 --ioengine=spdk --direct=1 \
--gtod_reduce=0 --cpumask=0xFFF0 --rate_iops=10k \
--rate_process=poisson \
--filename='trtype=RDMA adrfam=IPv4 traddr=1.1.79.1 trsvcid=4420 ns=1'
SPDK allocates the following entities for every work request in
receive queue (shared or not): reqs (1024 bytes), recvs (96 bytes),
cmds (64 bytes), cpls (16 bytes), in_capsule_buffer. All except the
last one are fixed size. In capsule data size is configured to 4096.
Memory consumption calculation (target):
- Multiple SRQ: core_num * ib_devs_num * SRQ_depth * (1200 +
in_capsule_data_size)
- Multiple RQ: queue_num * RQ_depth * (1200 + in_capsule_data_size)
We ignore admin queues in calculations for simplicity.
Cases:
1. Multiple SRQ with 1024 entries:
- Mem = 4 * 1 * 1024 * (1200 + 4096) = 20.7 MiB
(Constant number – does not depend on initiators number)
2. RQ with 128 entries for 64 initiators:
- Mem = 64 * 128 * (1200 + 4096) = 41.4 MiB
Results:
FIO_JOBS kIOPS Bandwidth,MiB/s AvgLatency,us MaxResidentSize,kiB
RQ SRQ RQ SRQ RQ SRQ RQ SRQ
1 8.623 8.623 33.7 33.7 13.89 14.03 144376 155624
2 17.3 17.3 67.4 67.4 14.03 14.1 145776 155700
4 34.5 34.5 135 135 14.15 14.23 146540 156184
8 69.1 69.1 270 270 14.64 14.49 148116 156960
16 138 138 540 540 14.84 15.38 151216 158668
32 276 276 1079 1079 16.5 16.61 157560 161936
64 513 502 2005 1960 1673 1612 170408 168440
128 535 526 2092 2054 3329 3344 195796 181524
256 571 571 2232 2233 6854 6873 246484 207856
We can see the benefit in memory consumption.
Change-Id: I40c70f6ccbad7754918bcc6cb397e955b09d1033
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/428458
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2018-10-04 14:59:08 +00:00
### NVMe-oF Target
Support for per-device shared receive queues in the RDMA transport has been added.
2019-03-05 23:46:20 +00:00
It is enabled by default for any device that supports it.
nvmf/rdma: Add shared receive queue support
This is a new feature for NVMEoF RDMA target, that is intended to save
resource allocation (by sharing them) and utilize the
locality (completions and memory) to get the best performance with
Shared Receive Queues (SRQs). We'll create a SRQ per core (poll
group), per device and associate each created QP/CQ with an
appropriate SRQ.
Our testing environment has 2 hosts.
Host 1:
CPU: Intel(R) Xeon(R) CPU E5-2609 0 @ 2.40GHz dual socket (8 cores total)
Network: ConnectX-5, ConnectX-5 VPI , 100GbE, single-port QSFP28, PCIe3.0 x16
Disk: Intel Optane SSD 900P Series
OS: Fedora 27 x86_64
Host 2:
CPU: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz dual-socket (24 cores total)
Network: ConnectX-4 VPI , 100GbE, dual-port QSFP28
Disk: Intel Optane SSD 900P Series
OS : CentOS 7.5.1804 x86_64
Hosts are connected via Spectrum switch.
Host 1 is running SPDK NVMeoF target.
Host 2 is used as initiator running fio with SPDK plugin.
Configuration:
- SPDK NVMeoF target: cpu mask 0x0F (4 cores), max queue depth 128,
max SRQ depth 1024, max QPs per controller 1024
- Single NVMf subsystem with single namespace backed by physical SSD disk
- fio with SPDK plugin: randread pattern, 1-256 jobs, block size 4k,
IO depth 16, cpu_mask 0xFFF0, IO rate 10k, rate process “poisson”
Here is a full fio command line:
fio --name=Job --stats=1 --group_reporting=1 --idle-prof=percpu \
--loops=1 --numjobs=1 --thread=1 --time_based=1 --runtime=30s \
--ramp_time=5s --bs=4k --size=4G --iodepth=16 --readwrite=randread \
--rwmixread=75 --randrepeat=1 --ioengine=spdk --direct=1 \
--gtod_reduce=0 --cpumask=0xFFF0 --rate_iops=10k \
--rate_process=poisson \
--filename='trtype=RDMA adrfam=IPv4 traddr=1.1.79.1 trsvcid=4420 ns=1'
SPDK allocates the following entities for every work request in
receive queue (shared or not): reqs (1024 bytes), recvs (96 bytes),
cmds (64 bytes), cpls (16 bytes), in_capsule_buffer. All except the
last one are fixed size. In capsule data size is configured to 4096.
Memory consumption calculation (target):
- Multiple SRQ: core_num * ib_devs_num * SRQ_depth * (1200 +
in_capsule_data_size)
- Multiple RQ: queue_num * RQ_depth * (1200 + in_capsule_data_size)
We ignore admin queues in calculations for simplicity.
Cases:
1. Multiple SRQ with 1024 entries:
- Mem = 4 * 1 * 1024 * (1200 + 4096) = 20.7 MiB
(Constant number – does not depend on initiators number)
2. RQ with 128 entries for 64 initiators:
- Mem = 64 * 128 * (1200 + 4096) = 41.4 MiB
Results:
FIO_JOBS kIOPS Bandwidth,MiB/s AvgLatency,us MaxResidentSize,kiB
RQ SRQ RQ SRQ RQ SRQ RQ SRQ
1 8.623 8.623 33.7 33.7 13.89 14.03 144376 155624
2 17.3 17.3 67.4 67.4 14.03 14.1 145776 155700
4 34.5 34.5 135 135 14.15 14.23 146540 156184
8 69.1 69.1 270 270 14.64 14.49 148116 156960
16 138 138 540 540 14.84 15.38 151216 158668
32 276 276 1079 1079 16.5 16.61 157560 161936
64 513 502 2005 1960 1673 1612 170408 168440
128 535 526 2092 2054 3329 3344 195796 181524
256 571 571 2232 2233 6854 6873 246484 207856
We can see the benefit in memory consumption.
Change-Id: I40c70f6ccbad7754918bcc6cb397e955b09d1033
Signed-off-by: Evgeniy Kochetov <evgeniik@mellanox.com>
Signed-off-by: Sasha Kotchubievsky <sashakot@mellanox.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/428458
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2018-10-04 14:59:08 +00:00
The size of a shared receive queue is defined by transport configuration file parameter
`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth` .
Default size is 4096.
2019-04-23 22:49:19 +00:00
Add model number as parameter to construct_nvmf_subsystem (-d option),
rather than using hardcoded define.
2019-04-22 08:37:26 +00:00
DIF passthrough feature has been added. DIF setting of the allocated bdevs is
exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is
passed through to the allocated bdevs.
2019-03-15 13:13:46 +00:00
### env
The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated.
For retrieving physical addresses, spdk_vtophys() should be used instead.
2019-04-30 16:55:16 +00:00
spdk_realloc() has been added to reallocate DMA/shared memory.
2019-03-23 21:38:14 +00:00
spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know
that device has a pending external hotremove request.
2019-03-11 12:14:19 +00:00
spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources
allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected
that common usage of those functions is to call them just before terminating the process.
2019-04-30 16:55:16 +00:00
Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA.
### DPDK
Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now.
2019-04-30 15:10:56 +00:00
Updated DPDK submodule to DPDK 19.02.
2019-03-21 22:02:04 +00:00
### rpc
New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application.
2019-01-15 12:52:11 +00:00
The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device
is specified, use that specified nbd device. If it's not specified, pick available one.
2019-04-22 08:40:23 +00:00
### iSCSI target
DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator.
DIF is attached into data for write I/O and stripped from data for read I/O.
2019-04-30 14:15:15 +00:00
### vhost
Added experimental support for running with the external, upstream rte_vhost library.
This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag.
The minimum supported rte_vhost version (DPDK version) is 19.05-rc1.
As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and
fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks
if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting.
The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first
unoccupied target id will be used.
2019-04-30 14:16:06 +00:00
### AIO
AIO bdev module can now reap I/O completions directly from userspace, significantly improving
the overall performance.
2019-04-30 15:14:22 +00:00
### blobfs
Synchronous IO operations no longer use spdk_io_channel, but instead use
spdk_fs_thread_ctx. The behavior is otherwise identical.
### OCF
Added support for caching multiple bdevs using a single bdev as a cache.
### notify
Added the notify library that provides a high performance local event bus
between libraries. Example usage was added to bdev module, which reports
notifications for added and removed bdevs via RPC.
### sock
Added new API spdk_sock_readv() to the sock library for performing vectored
reads.
### event
The function spdk_subsystem_init() no longer requires spdk_event as an argument.
Changed API of spdk_subsystem_config_json() to no longer be asynchronous.
2019-04-30 17:02:36 +00:00
### io_uring
A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel
API was added. This module requires liburing.
2019-04-30 17:09:37 +00:00
### build
Options to easily compile with profile guided optimization have been added to
`configure` . To use profile guided optimization with SPDK, run
`./configure --with-pgo-capture` , build SPDK, then run a workload of your
choosing. Then, simply run `./configure --with-pgo-enable` and recompile to
build using the generated profile data. Profile guided optimization can yield
very large performance improvements, especially on GCC 8 and clang 7. This may
be combined with link time optimization which has been available under the
`--enable-lto` configure option for several releases.
2019-04-30 15:45:12 +00:00
### compression bdev/reduce library
Added "reduce" block compression scheme based on using SSDs for storing
compressed blocks of storage and presistent memory for metadata. Please see
[compression ](https://spdk.io/doc/bdev.html ) for more details.
2019-01-29 09:20:39 +00:00
## v19.01:
2018-10-29 12:34:15 +00:00
2019-01-29 13:11:15 +00:00
### ocf bdev
New virtual bdev module based on [Open CAS Framework ](https://open-cas.github.io/ ) has been added.
This module allows for the use of one bdev to act as a high performance cache in front of another bdev.
Please see [documentation ](https://spdk.io/doc/bdev.html#bdev_config_cas ) for more details.
Only write through mode is currently supported and this feature is considered experimental.
2019-01-08 19:40:06 +00:00
### event framework
For `spdk_app_parse_args` , add return value to the callback which parses application
specific command line parameters to protect SPDK applications from crashing by invalid
values from user input.
2019-01-30 10:18:50 +00:00
By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved
memory size can be still set with `-s` or `--mem-size` option, although the default value
was reduced down to 0.
2019-01-29 13:11:15 +00:00
A custom hugetlbfs directory can now be specified via spdk_app_opts.
This can be used to configure hugepages with different sizes, a different size limit,
or different access permissions than the system's default hugepage pool.
SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option.
2018-12-13 15:54:01 +00:00
### environment
2019-01-29 20:45:10 +00:00
spdk_vtophys() has been refactored to accept length of the translated region as a new
parameter. The function will now update that parameter with the largest possible value
for which the memory is contiguous in the physical memory address space.
The following functions were removed:
- spdk_pci_nvme_device_attach()
- spdk_pci_nvme_enumerate()
- spdk_pci_ioat_device_attach()
- spdk_pci_ioat_enumerate()
- spdk_pci_virtio_device_attach()
- spdk_pci_virtio_enumerate()
They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which
require a new spdk_pci_driver object to be provided. It can be one of the following:
- spdk_pci_nvme_get_driver()
- spdk_pci_ioat_get_driver()
- spdk_pci_virtio_get_driver()
spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual
spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for
each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach()
or spdk_pci_enumerate() like any other device.
A new spdk_pause() function was added to pause CPU execution for an implementation specific
amount of time. Quoting from DPDK function this is based on: "This call is intended for
tight loops which poll a shared resource or wait for an event. A short pause within the loop
may reduce the power consumption."
2018-12-13 15:54:01 +00:00
A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init
is added into it. If user is using DPDK, and already called rte_eal_init, then include
include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init.
2019-01-21 09:05:59 +00:00
ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture
to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L
by explicitly passing --without-isal to the configure script.
2018-12-06 04:20:51 +00:00
### util
A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid.
2018-12-13 23:05:23 +00:00
An new parameter `init_crc` representing the initial CRC value was added to
`spdk_crc16_t10dif` . The parameter can be used to calculate a CRC value spanning
multiple separate buffers.
2019-01-16 02:28:14 +00:00
New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX
formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF
insert and strip.
2019-01-29 13:11:15 +00:00
Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol`
and `strtoll` .
Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string
with automatic buffer re-allocation.
2018-10-15 06:20:35 +00:00
### nvme
2019-01-02 20:20:21 +00:00
Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are
introduced to support further security protocol development.
2018-10-15 06:20:35 +00:00
admin_timeout_ms was added to NVMe controller initialization options, users
can change the default value when probing a controller.
2018-11-30 01:46:55 +00:00
Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts,
it will be used to enable the digest support for the NVMe/TCP transport.
Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With
this new transport, it can be used to connect the NVMe-oF target with the
same TCP/IP support.
2018-12-05 21:57:53 +00:00
Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr
arg refers to a Discovery Controller or not.
2018-12-04 22:09:31 +00:00
Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id`
for parsing host address and host service ID arguments on a per connection basis.
2018-12-04 23:30:11 +00:00
The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to
use when connecting to the controller backing the NVMe bdev.
2018-10-19 20:19:09 +00:00
### NVMe-oF Target
The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts` .
Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen`
will also no longer implicitly initialize a transport with the default target options (since
there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create`
prior to calling `spdk_nvmf_tgt_listen` .
Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to
`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit.
2018-08-02 02:21:45 +00:00
2019-01-29 13:11:15 +00:00
Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts,
and also updated the related rpc function nvmf_create_transport, to make this
2018-12-14 14:29:48 +00:00
configurable parameter available to users. The `num_shared_buffers` is used to
configure the shared buffer numbers of the transport used by RDMA or TCP transport.
2019-01-29 13:11:15 +00:00
`buf_cache_size` configures number of shared buffers to cache per poll group.
2018-12-14 14:29:48 +00:00
2018-08-02 02:21:45 +00:00
### nvmf
Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this tranport,
the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF
protocol via TCP/IP from the host.
2018-10-19 20:19:09 +00:00
2019-01-29 13:11:15 +00:00
Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory.
By default, the RDMA transport will use the ibverbs library to create protection domains and register memory.
Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration.
2018-11-06 12:02:41 +00:00
### bdev
2019-01-29 13:11:15 +00:00
Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev.
Please see [documentation ](https://spdk.io/doc/bdev.html#rpc_bdev_histogram ) for more details.
Added `required_alignment` field to `spdk_bdev` , that specifies an alignment requirement for data buffers associated with an spdk_bdev_io.
Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module.
2018-11-06 12:02:41 +00:00
On shutdown, bdev unregister now proceeds in top-down fashion, with
claimed bdevs skipped (these will be unregistered later, when virtual
bdev built on top of the respective base bdev unclaims it). This
allows virtual bdevs to be shut down cleanly as opposed to the
previous behavior that didn't differentiate between hotremove and
planned shutdown.
2019-01-30 10:54:33 +00:00
The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC
for the bdev type to be removed (i.e. delete_malloc_bdev).
2019-01-29 13:11:15 +00:00
Added support for separate bandwidth rate limits for read and write to QoS in bdev layer.
Bdev I/O statistics now track unmap opertations.
2019-01-15 15:31:05 +00:00
### logical volumes
Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC.
This allows for basing clones on top of lvol_bdev without first creating a snapshot.
2019-01-29 10:23:26 +00:00
Added option to change method for data erasure when deleting lvol or resizing down.
Default of unmapping clusters can now be changed to writing zeroes or no operation.
2019-02-28 09:57:19 +00:00
Added option to change method for erasing data region on lvol store creation.
Default of unmapping can now be changed to writing zeroes or no operation.
2018-11-27 07:26:59 +00:00
### log
"trace flags" are now referred to as "log flags" in the SPDK log API. The
set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated,
and set_log_flag, clear_log_flag and get_log_flags RPCs have been added.
2018-11-29 11:12:26 +00:00
### trace
2019-01-30 11:19:23 +00:00
New `trace_record` application was added. It can be used to poll spdk trace shm file and
append any new trace entries into another specified file. This can help retain those entries
that would otherwise be overwritten in the shm file. See
[Capturing sufficient trace events ](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events )
for more details.
2019-01-29 13:11:15 +00:00
Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app
with argument "--num-trace-entries < NUM > " provided.
2018-11-29 11:12:26 +00:00
New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and
each tpoint group status.
New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or
disable a specific tpoint group.
2019-01-23 15:52:08 +00:00
### ftl
EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as
block devices. The module is split into the library (located in lib/ftl) and bdev_ftl
(lib/bdev/ftl). See the [documentation ](https://spdk.io/doc/ftl.html ) for more details.
2019-01-29 20:50:18 +00:00
### vhost
2019-02-01 03:11:19 +00:00
A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious
vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which
would result in a partial denial of service in the SPDK vhost target. These types of descriptor
chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted
vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny
Yakovlev.)
2019-01-29 20:50:18 +00:00
Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file.
Each connection (internally called a vhost session) will have access to the same storage, but
will use different virtqueues, different features and possibly different memory.
2019-01-29 13:11:15 +00:00
### vhost scsi
SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated.
Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes.
2019-01-30 08:48:40 +00:00
### DPDK
2019-01-29 21:01:48 +00:00
DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new
multi-process device hotplug yet and continues to work the same way it always did.
2019-01-30 08:48:40 +00:00
Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now.
2019-01-30 10:54:33 +00:00
### RPC
The following RPC commands deprecated in the previous release are now removed:
- construct_virtio_user_scsi_bdev
- construct_virtio_pci_scsi_bdev
- construct_virtio_user_blk_bdev
- construct_virtio_pci_blk_bdev
- remove_virtio_scsi_bdev
- construct_nvmf_subsystem
2019-01-30 10:55:19 +00:00
### Miscellaneous
The configure options `--with-raid` and `--without-raid` that were deprecated in the previous
release are now removed.
2019-01-29 13:11:15 +00:00
### nbd
Starting nbd using `spdk_nbd_start` is now performed asynchronously.
### net framework
Net framework initialization and finish is now done asynchronously.
### rpc
Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state.
Added `spdk_rpc_get_state` to check current state of RPC server.
RPC `wait_subsystem_init` has been added to allow clients to block untill all subsystems are initialized.
### json rpc
JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll.
JSON RPC server can now recieve a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb`
and `spdk_jsonrpc_conn_del_close_cb` .
2018-10-29 12:31:37 +00:00
## v18.10:
2018-08-02 17:32:39 +00:00
2018-08-15 16:56:25 +00:00
### nvme
2018-09-25 20:07:32 +00:00
spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive()
were added to support sending or receiving security protocol data to or from
nvme controller.
2018-08-15 16:56:25 +00:00
spdk_nvme_ns_get_extended_sector_size() was added. This function includes
the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still
returns only the data size per sector, not including metadata.
2018-10-29 14:23:24 +00:00
New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller.
See the [send_nvme_cmd ](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd ) documentation
for more details.
2018-08-14 17:56:26 +00:00
### Build System
New `configure` options, `--with-shared` and `--without-shared`
[default], provide the capability to build, or not, SPDK shared libraries.
This includes the single SPDK shared lib encompassing all of the SPDK
static libs as well as individual SPDK shared libs corresponding to
each of the SPDK static ones. Although the production of the shared
libs conforms with conventional version naming practices, such naming
does not at this time confer any SPDK ABI compatibility claims.
2018-08-14 08:39:27 +00:00
### bdev
spdk_bdev_alias_del_all() was added to delete all alias from block device.
2018-03-07 23:44:06 +00:00
A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev
Framework. The module initially uses a software AESNI CBC cipher with experimental support for the
Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work
2018-10-31 14:50:11 +00:00
may include additional ciphers as well as consideration for authentication.
2018-03-07 23:44:06 +00:00
2018-09-18 22:55:06 +00:00
The RAID virtual bdev module is now always enabled by default. The configure --with-raid and
--without-raid options are now ignored and deprecated and will be removed in the next release.
2018-06-22 02:15:02 +00:00
Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer.
See the new [set_bdev_qos_limit ](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit )
documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at
18.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both
bandwidth and IOPS limits.
2018-10-29 16:11:11 +00:00
spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed.
2018-09-06 20:07:42 +00:00
### Environment Abstraction Layer and Event Framework
The size parameter of spdk_mem_map_translate is now a pointer. This allows the
function to report back the actual size of the translation relative to the original
request made by the user.
2018-09-11 22:03:31 +00:00
A new structure spdk_mem_map_ops has been introduced to hold memory map related
callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc
in lieu of the notify callback.
2018-10-29 07:02:30 +00:00
### DPDK 18.08
The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically
utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1.
Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup,
but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try
to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using
only as many hugepages as it is really needed.
Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed
to be physically contiguous.
2018-10-30 10:38:39 +00:00
### I/OAT
I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors.
2018-10-18 11:29:31 +00:00
### iscsi target
2018-08-22 02:30:57 +00:00
Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC
method for CHAP authentication in discovery sessions have been changed to
align with `construct_target_node` RPC method. Old names are still usable
but will be removed in future release.
2018-08-20 00:12:59 +00:00
`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have
been added to set CHAP authentication for discovery sessions and existing
target nodes, respectively.
2018-08-22 04:43:18 +00:00
2018-08-27 23:37:35 +00:00
The SPDK iSCSI target supports an AuthFile which can be used to load CHAP
shared secrets when the iSCSI target starts. SPDK previously provided a
default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was
specified. This default has been removed. Users must now explicitly specify
the location of this file to load CHAP shared secrets from a file, or use
the related iSCSI RPC methods to add them at runtime.
2018-08-22 06:09:05 +00:00
2018-10-18 11:29:31 +00:00
### iscsi initiator
2018-10-18 22:07:38 +00:00
The SPDK iSCSI initiator is no longer considered experimental and becomes
2018-10-18 11:29:31 +00:00
a first-class citizen among bdev modules. The basic usage has been briefly
described in the bdev user guide: [iSCSI bdev ](https://spdk.io/doc/bdev.html#bdev_config_iscsi )
2018-08-28 23:36:46 +00:00
### Miscellaneous
2018-10-30 10:30:52 +00:00
The SPDK application framework can now parse long name command line parameters.
Most single-character parameters have a long name equivalent now. See the
[Command Line Parameters ](https://spdk.io/doc/app_overview.html ) documentation
for details or use the `--help` command line parameter to list all available
params.
bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already
used by existing apps for memory size.
bdevio can now accept all SPDK command line parameters. The config now has to
be provided with `-c` or `--config` param.
The following ioat/perf and nvme/perf parameters were renamed as well:
`-s` (io size) to `-o`
`-d` (mem size) to `-s`
2018-08-28 23:36:46 +00:00
The ReactorMask config file parameter has been deprecated. Users should
use the -m or --cpumask command line option to specify the CPU core mask
for the application.
2018-09-04 20:58:35 +00:00
Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt
and vhost. Config file pathnames may now only be specified using the
-c command line option.
2018-09-06 21:09:14 +00:00
Users may no longer set DPDK_DIR in their environment to specify the
location of the DPDK installation used to build SPDK. Using DPDK_DIR
has not been the documented nor recommended way to specify the DPDK
location for several releases, but removing it ensures no unexpected
surprises for users who may have DPDK_DIR defined for other reasons.
Users should just use the "configure" script to specify the DPDK
location before building SPDK.
2018-09-11 13:26:14 +00:00
Although we know that many developers still use Python 2 we are officially
switching to Python3 with requirement that all new code must be valid also
for Python 2 up to the EOL which is year 2020.
Invoking interpreter explicitly is forbidden for executable scripts. There
is no need to use syntax like "python ./scripts/rpc.py". All executable
scripts must contain proper shebang pointing to the right interpreter.
Scripts without shebang musn't be executable.
2018-07-26 01:32:05 +00:00
A Python script has been added to enable conversion of old INI config file
to new JSON-RPC config file format. This script can be found at
scripts/config_converter.py. Example how this script can be used:
~~~{.sh}
cat old_format.ini | scripts/config_converter.py > new_json_format.json
~~~
2018-10-08 01:51:03 +00:00
### Sock
Two additional parameters were added to spdk_sock_get_addr() for the server
port and client port. These parameters are named "sport" and "cport"
respectively.
2018-10-18 12:25:17 +00:00
### Virtio
The following RPC commands have been deprecated:
- construct_virtio_user_scsi_bdev
- construct_virtio_pci_scsi_bdev
- construct_virtio_user_blk_bdev
- construct_virtio_pci_blk_bdev
2018-10-18 13:12:03 +00:00
- remove_virtio_scsi_bdev
2018-10-18 12:25:17 +00:00
2018-10-18 13:12:03 +00:00
The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev`
command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev`
was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI
devices.
2018-10-18 12:25:17 +00:00
2018-10-29 14:53:45 +00:00
### Blobfs
spdk_file_get_id() returning unique ID for the file was added.
2018-10-29 16:20:48 +00:00
### JSON
2018-10-29 16:54:24 +00:00
Added jsonrpc-client C library intended for issuing RPC commands from applications.
2018-10-29 16:20:48 +00:00
Added API enabling iteration over JSON object:
- spdk_json_find()
- spdk_json_find_string()
- spdk_json_find_array()
- spdk_json_object_first()
- spdk_json_array_first()
- spdk_json_next()
2018-10-29 12:46:23 +00:00
### Blobstore
Blobstore I/O operations are now based on io_units, instead of blobstore page size.
The io_unit size is now the same as the underlying block device's block size.
Logical volumes built on a block device with 512B block size can now be used as boot devices
in QEMU.
2018-10-29 12:38:16 +00:00
### SPDKCLI
The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental.
Support for the iSCSI and NVMe-oF targets has been added.
2018-08-02 17:29:48 +00:00
## v18.07:
2018-04-27 23:16:47 +00:00
2018-07-31 16:57:23 +00:00
### bdev
A new public header file bdev_module.h has been introduced to facilitate the
development of new bdev modules. This header includes an interface for the
spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of
multiple virtual bdevs on top of a single base bdev and should act as the
primary API for module authors.
spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide
options.
A mechanism for handling out of memory condition errors (ENOMEM) returned from
I/O submission requests at the bdev layer has been added. See
spdk_bdev_queue_io_wait().
The spdk_bdev_get_io_stat() function now returns cumulative totals instead of
resetting on each call. This allows multiple callers to query I/O statistics
without conflicting with each other. Existing users will need to adjust their
code to record the previous I/O statistics to calculate the delta between calls.
I/O queue depth tracking and samples options have been added. See
spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and
spdk_bdev_set_qd_sampling_period().
2018-05-08 11:30:29 +00:00
### RAID module
A new bdev module called "raid" has been added as experimental module which
2018-07-31 16:57:23 +00:00
aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note
that vhost will not work with this module because it does not yet have support
for multi-element io vectors.
2018-05-08 11:30:29 +00:00
2018-06-26 21:54:37 +00:00
### Log
2018-07-31 16:57:23 +00:00
The debug log component flag available on several SPDK applications has been
renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the
option to be added to tools that already use `-t` to mean something else.
### Blobstore
A new function, spdk_bs_dump(), has been added that dumps all of the contents of
a blobstore to a file pointer. This includes the metadata and is very useful for
debugging.
Two new operations have been added for thin-provisioned blobs.
spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions
of the blob and populate them with the correct data by reading from the backing
blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate
clusters that correspond to data in the blob's immediate parent. Clusters
allocated to grandparents or that aren't allocated at all will remain
thin-provisioned.
### BlobFS
Changed the return type of spdk_file_truncate() from void to int to allow the
propagation of `ENOMEM` errors.
2018-06-26 21:54:37 +00:00
2018-05-17 17:32:42 +00:00
### NVMe Driver
2018-07-31 16:57:23 +00:00
The new API functions spdk_nvme_qpair_add_cmd_error_injection() and
spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error
emulation. Users can set a specified command to fail with a particular error
status.
Changed the name `timeout_sec` parameter to `timeout_us` in
spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from
uint32_t to uint64_t. This will give users more fine-grained control over the
timeout period.
2018-05-17 17:32:42 +00:00
2018-07-31 16:57:23 +00:00
Basic support for Open Channel SSDs was added. See nvme_ocssd.h
### NVMe Over Fabrics
The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback
as a parameter.
spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs.
spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed
number of namespaces for a given subsystem.
2018-07-16 01:58:45 +00:00
2018-06-11 19:40:29 +00:00
### Build System
2018-07-31 16:57:23 +00:00
The build system now generates a combined shared library (libspdk.so) that may
be used in place of the individual static libraries (libspdk_*.a). The combined
library includes all components of SPDK and is intended to make linking against
SPDK easier. The static libraries are also still provided for users that prefer
to link only the minimal set of components required.
### git pre-commit and pre-push hooks
2018-06-11 19:40:29 +00:00
2018-07-31 16:57:23 +00:00
The pre-commit hook will run `scripts/check_format.sh` and verify there are no
formating errors before allowing `git commit` to run. The pre-push hook runs
`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and
clang compiler before allowing `git push` to run. Following each DEBUG build
`test/unit/unittest.sh` is run and verified. Results are recorded in the
`make.log` file.
To enable type: 'git config core.hooksPath .githooks'. To override after
configuration use the `git --no-verify` flag.
2018-03-22 23:24:14 +00:00
2018-06-05 19:31:50 +00:00
### RPC
The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node
rather than always returning `true` .
2018-07-31 16:57:23 +00:00
### DPDK 18.05
2018-05-23 18:30:01 +00:00
2018-07-31 16:57:23 +00:00
The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports
dynamic memory allocation, but due to some issues found after the DPDK 18.05 release,
that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use
the legacy memory allocation model. The plan is to enable dynamic memory allocation
after the DPDK 18.08 release which should fix these issues.
2018-06-28 17:37:00 +00:00
2018-07-31 16:57:23 +00:00
### Environment Abstraction Layer and Event Framework
2018-05-31 21:44:49 +00:00
The spdk_mem_map_translate() function now takes a size parameter to indicate the size of
the memory region. This can be used by environment implementations to validate the
requested translation.
2018-06-11 18:49:43 +00:00
The I/O Channel implementation has been moved to its own library - lib/thread. The
public API that was previously in spdk/io_channel.h is now in spdk/thread.h The
file spdk/io_channel.h remains and includes spdk/thread.h.
2018-07-31 16:57:23 +00:00
spdk_reactor_get_tsc_stats was added to return interesting statistics for each
reactor.
2018-02-05 18:15:54 +00:00
2018-06-05 23:33:20 +00:00
### IOAT
IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable
option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is
now deprecated and will be removed in a future release.
2018-04-26 19:46:22 +00:00
## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP
2018-02-02 17:39:35 +00:00
2018-04-23 17:34:45 +00:00
### vhost
The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the
DPDK rte_vhost vulnerability [CVE-2018-1059 ](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059 ).
Please see this [security advisory ](https://access.redhat.com/security/cve/cve-2018-1059 )
for additional information on the DPDK vulnerability.
2018-04-26 19:46:22 +00:00
Workarounds have been added to ensure vhost compatibility with QEMU 2.12.
EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the
[vhost documentation ](http://www.spdk.io/doc/vhost.html ) for more details.
### Unified Target Application
A new unified SPDK target application, `spdk_tgt` , has been added. This application combines the
functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target,
and vhost target. The new application can be managed through the existing configuration file and
[JSON-RPC ](http://www.spdk.io/doc/jsonrpc.html ) methods.
2018-04-23 17:34:45 +00:00
### Env
2018-04-18 23:25:05 +00:00
2018-04-26 19:46:22 +00:00
spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk().
2018-04-18 23:25:05 +00:00
2018-02-23 06:43:28 +00:00
New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added.
These new functions have a `flags` parameter that allows the user to specify whether the allocated
memory needs to be suitable for DMA and whether it should be shared across processes with the same
shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will
eventually be deprecated and removed.
2018-04-10 21:53:20 +00:00
### Bdev
2018-04-26 19:46:22 +00:00
A new optional bdev module interface function, `init_complete` , has been added to notify bdev modules
when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require
2018-04-10 21:53:20 +00:00
notification that the set of initialization examine() calls is complete.
2018-04-26 19:46:22 +00:00
The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with
the spdk_bdev_get_uuid() function.
Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the
[set_bdev_qos_limit_iops ](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops ) documentation
for more details.
2018-02-02 17:39:35 +00:00
### RPC
2018-04-26 19:46:22 +00:00
The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed.
2018-02-02 17:39:35 +00:00
Users should switch to the `-r` command-line parameter instead.
2018-02-14 17:12:50 +00:00
The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as
needed; previously, the response was limited to 32 kilobytes.
2018-04-24 11:11:04 +00:00
### SPDKCLI
EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available.
See the [SPDKCLI ](http://www.spdk.io/doc/spdkcli.html ) documentation for more details.
2017-04-21 23:35:11 +00:00
### NVMe Driver
2018-04-26 19:46:22 +00:00
EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is
2017-04-21 23:35:11 +00:00
experimental pending a functional allocator to free and reallocate CMB buffers.
2018-02-02 17:39:35 +00:00
2018-03-22 00:36:06 +00:00
spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available.
2018-04-26 19:46:22 +00:00
New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns()
have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if
a namespace ID is active.
2018-02-23 14:38:24 +00:00
2018-02-13 00:03:01 +00:00
### NVMe-oF Target
2018-04-26 19:46:22 +00:00
Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters
to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes
the backing bdev's UUID as the namespace UUID when available.
2018-02-13 00:03:01 +00:00
2018-04-26 19:46:22 +00:00
spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion.
2018-04-18 16:31:40 +00:00
2018-02-21 21:37:37 +00:00
### Blobstore
A number of functions have been renamed:
- spdk_bs_io_write_blob() => spdk_blob_io_write()
- spdk_bs_io_read_blob() => spdk_blob_io_read()
- spdk_bs_io_writev_blob() => spdk_blob_io_writev()
- spdk_bs_io_readv_blob() => spdk_blob_io_readv()
- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap()
- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes()
The old names still exist but are deprecated. They will be removed in the v18.07 release.
2018-03-19 21:05:44 +00:00
spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O
are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still
required to sync the updated metadata to disk.
2018-04-11 22:12:13 +00:00
### Logical Volumes
A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added.
2018-04-24 11:11:04 +00:00
Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination.
2018-08-27 08:42:35 +00:00
New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols
2018-04-24 11:11:04 +00:00
and Clones of existing Snapshots.
See the [lvol snapshots ](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots ) documentation
for more details.
2018-04-26 19:46:22 +00:00
Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method.
2018-02-22 23:50:51 +00:00
### Lib
2018-04-26 19:46:22 +00:00
A set of changes were made in the SPDK's lib code altering
2018-02-22 23:50:51 +00:00
instances of calls to `exit()` and `abort()` to return a failure instead
wherever reasonably possible.
spdk_app_start() no longer exit()'s on an internal failure, but
instead returns a non-zero error status.
spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid
option, but instead returns SPDK_APP_PARSE_ARGS_HELP and
SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS
on success.
2018-03-28 22:50:52 +00:00
spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07.
2018-03-13 00:16:47 +00:00
### I/O Channels
The prototype for spdk_poller_fn() has been modified; it now returns a value indicating
whether or not the poller did any work. Existing pollers will need to be updated to
return a value.
2018-04-26 19:46:22 +00:00
### iSCSI Target
The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace
TCP/IP stack. See the [iSCSI VPP documentation ](http://www.spdk.io/doc/iscsi.html#vpp ) for more
details.
2017-12-04 17:07:04 +00:00
### iSCSI initiator
An iSCSI initiator bdev module has been added to SPDK. This module should be considered
experimental pending additional features and tests. More details can be found in
lib/bdev/iscsi/README.
2018-04-03 23:07:39 +00:00
### PMDK
2018-04-26 19:46:22 +00:00
The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of
`--with-nvml` . This reflects the renaming of the persistent memory library from NVML to
2018-04-03 23:07:39 +00:00
PMDK.
2018-04-24 10:17:56 +00:00
### Virtio Block driver
A userspace driver for Virtio Block devices has been added. It was built on top of the
[Virtio ](http://www.spdk.io/doc/virtio.html ) library and can be managed similarly to
the Virtio SCSI driver. See the
[Virtio Block ](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk ) reference for
more information.
2018-04-24 11:49:13 +00:00
### Virtio with 2MB hugepages
2018-04-26 19:46:22 +00:00
The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option
2018-04-24 11:49:13 +00:00
enables SPDK Virtio to work with 2MB hugepages.
See [2MB hugepages ](http://www.spdk.io/doc/virtio.html#virtio_2mb ) for details.
2018-01-30 21:22:20 +00:00
## v18.01: Blobstore Thin Provisioning
2017-11-15 17:36:36 +00:00
### Build System
The build system now includes a `make install` rule, including support for the common
`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix
may be set via the configure `--prefix` option. Example: `make install prefix=/usr` .
2017-11-09 23:33:29 +00:00
### RPC
A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/run/spdk.sock.
A -r option command line option has been added to enable an alternative UNIX domain socket location,
or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file
section is now deprecated and will be removed in the v18.04 release.
2017-11-14 20:42:17 +00:00
### I/O Channels
spdk_poller_register() and spdk_poller_unregister() were moved from the event
framework (include/spdk/event.h) to the I/O channel library
(include/spdk/io_channel.h). This allows code that doesn't depend on the event
framework to request registration and unregistration of pollers.
2017-12-11 22:14:19 +00:00
spdk_for_each_channel() now allows asynchronous operations during iteration.
Instead of immediately continuing the interation upon returning from the iteration
callback, the user must call spdk_for_each_channel_continue() to resume iteration.
2017-11-17 21:49:36 +00:00
### Block Device Abstraction Layer (bdev)
The poller abstraction was removed from the bdev layer. There is now a general purpose
abstraction for pollers available in include/spdk/io_channel.h
2017-12-18 19:57:01 +00:00
### Lib
A set of changes were made in the SPDK's lib code altering,
instances of calls to `exit()` and `abort()` to return a failure instead
wherever reasonably possible. This has resulted in return type changes of
the API for:
- spdk_env_init() from type `void` to `int` .
- spdk_mem_map_init() from type `void` to `int` .
Applications making use of these APIs should be modified to check for
a non-zero return value instead of relying on them to fail without return.
2017-12-07 00:59:17 +00:00
### NVMe Driver
2017-12-12 06:25:32 +00:00
SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind:
Only physical removal events are supported; removing devices via the sysfs `remove` file will not work.
2017-12-07 00:59:17 +00:00
2017-11-20 18:36:36 +00:00
### NVMe-oF Target
Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available
cores round-robin. The "Core" option in the configuration file has been removed.
2017-12-08 16:38:33 +00:00
### Blobstore
A number of functions have been renamed:
2018-01-05 18:24:03 +00:00
- spdk_bs_md_resize_blob() => spdk_blob_resize()
- spdk_bs_md_sync_blob() => spdk_blob_sync_md()
- spdk_bs_md_close_blob() => spdk_blob_close()
- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names()
- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value()
- spdk_blob_md_set_xattr() => spdk_blob_set_xattr()
- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr()
- spdk_bs_md_create_blob() => spdk_bs_create_blob()
- spdk_bs_md_open_blob() => spdk_bs_open_blob()
- spdk_bs_md_delete_blob() => spdk_bs_delete_blob()
- spdk_bs_md_iter_first() => spdk_bs_iter_first()
- spdk_bs_md_iter_next() => spdk_bs_iter_next()
The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument
2017-12-12 20:42:56 +00:00
rather than struct spdk_blob ** .
2018-01-05 18:24:03 +00:00
The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument
2017-12-12 20:53:10 +00:00
rather than struct spdk_blob ** .
2018-01-30 21:22:20 +00:00
Thin provisioning support has been added to the blobstore. It can be enabled by setting the
`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext().
2017-12-12 09:53:18 +00:00
### NBD device
The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be
achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method.
See the [GPT ](http://www.spdk.io/doc/bdev.html#bdev_config_gpt ) documentation for more details.
2018-01-11 13:35:35 +00:00
### FIO plugin
2018-08-27 08:42:35 +00:00
SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped,
2018-01-11 13:35:35 +00:00
although it still remains to work for now. The new FIO contains huge amount of bugfixes and
it's recommended to do an update.
2018-01-12 14:21:48 +00:00
### Virtio library
Previously a part of the bdev_virtio module, now a separate library. Virtio is now available
via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing
new Virtio backends, namely Virtio-BLK.
2018-01-19 18:15:38 +00:00
### iSCSI
The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated
to an epoll/kqueue descriptor on the master core when idle.
2017-10-23 18:07:48 +00:00
## v17.10: Logical Volumes
2017-06-06 17:24:07 +00:00
2017-10-23 18:07:48 +00:00
### New dependencies
2017-09-14 14:53:36 +00:00
2017-10-23 18:07:48 +00:00
libuuid was added as new dependency for logical volumes.
libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08.
2017-09-14 14:53:36 +00:00
2017-08-17 18:45:43 +00:00
### Block Device Abstraction Layer (bdev)
2017-08-17 14:18:52 +00:00
2017-06-06 17:24:07 +00:00
An [fio ](http://github.com/axboe/fio ) plugin was added that can route
2017-10-23 18:07:48 +00:00
I/O to the bdev layer. See the [plugin documentation ](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/ )
2017-06-06 17:24:07 +00:00
for more information.
2017-07-19 21:32:04 +00:00
spdk_bdev_unmap() was modified to take an offset and a length in bytes as
arguments instead of requiring the user to provide an array of SCSI
unmap descriptors. This limits unmaps to a single contiguous range.
2017-10-23 18:07:48 +00:00
spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out.
If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using
write commands.
2017-08-01 18:28:29 +00:00
2017-08-28 21:55:35 +00:00
New API functions that accept I/O parameters in units of blocks instead of bytes
have been added:
- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks()
- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks()
- spdk_bdev_write_zeroes_blocks()
- spdk_bdev_unmap_blocks()
2017-10-23 18:07:48 +00:00
The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be
retried later.
2017-08-10 00:28:32 +00:00
### Linux AIO bdev
The AIO bdev now allows the user to override the auto-detected block size.
2017-08-07 17:17:29 +00:00
### NVMe driver
The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field.
NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take
into account when splitting I/O requests.
2017-08-17 18:45:43 +00:00
The HotplugEnable option in `[Nvme]` sections of the configuration file is now
"No" by default. It was previously "Yes".
2017-10-23 18:07:48 +00:00
The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the
2017-08-21 17:32:22 +00:00
NVMe Controller associated with a given namespace.
2017-09-27 17:52:47 +00:00
The NVMe library now allows the user to specify a host identifier when attaching
to a controller. The host identifier is used as part of the Reservations feature,
as well as in the NVMe-oF Connect command. The default host ID is also now a
randomly-generated UUID, and the default host NQN uses the host ID to generate
a UUID-based NQN.
2017-10-23 18:07:48 +00:00
spdk_nvme_connect() was added to allow the user to connect directly to a single
NVMe or NVMe-oF controller.
2017-08-30 20:21:12 +00:00
### NVMe-oF Target (nvmf_tgt)
2017-08-15 21:55:41 +00:00
2017-10-23 18:07:48 +00:00
The NVMe-oF target no longer requires any in-capsule data buffers to run, and
the feature is now entirely optional. Previously, at least 4 KiB in-capsule
2017-08-15 21:55:41 +00:00
data buffers were required.
2017-08-30 20:21:12 +00:00
NVMe-oF subsytems have a new configuration option, AllowAnyHost, to control
whether the host NQN whitelist is enforced when accepting new connections.
If no Host options have been specified and AllowAnyHost is disabled, the
connection will be denied; this is a behavior change from previous releases,
which allowed any host NQN to connect if the Host list was empty.
AllowAnyHost is disabled by default.
2017-08-30 23:55:48 +00:00
NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number
of namespaces per subsystem is no longer limited.
2017-09-06 00:26:14 +00:00
The NVMe-oF target now supports the Write Zeroes command.
2017-08-16 17:05:24 +00:00
### Environment Abstraction Layer
A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide
additional clarity when constructing spdk_mempools. Previously, -1 could be
passed and the library would choose a reasonable default, but this new value
makes it explicit that the default is being used.
2017-08-18 16:41:26 +00:00
### Blobstore
2017-10-23 18:07:48 +00:00
The blobstore super block now contains a bstype field to identify the type of the blobstore.
Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load().
2017-10-20 12:52:19 +00:00
spdk_bs_destroy() was added to allow destroying blobstore on device
with an initialized blobstore.
2017-08-18 16:41:26 +00:00
spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable
scattered payloads.
2017-08-21 17:32:22 +00:00
2017-10-23 18:07:48 +00:00
A CLI tool for blobstore has been added, allowing basic operations through either command
line or shell interface. See the [blobcli ](https://github.com/spdk/spdk/tree/master/examples/blob/cli )
documentation for more details.
2017-09-29 20:11:15 +00:00
2017-08-22 20:06:57 +00:00
### Event Framework
The ability to set a thread name, previously only used by the reactor code, is
2017-10-23 18:07:48 +00:00
now part of the spdk_thread_allocate() API. Users may specify a thread name
2017-08-22 20:06:57 +00:00
which will show up in tools like `gdb` .
2017-09-06 20:17:54 +00:00
### Log
2017-10-23 18:07:48 +00:00
The spdk_trace_dump() function now takes a new parameter to allow the caller to
specify an output file handle (stdout or stderr, for example).
2017-08-22 20:06:57 +00:00
2017-09-14 14:53:36 +00:00
### Logical Volumes
Logical volumes library built on top of SPDK blobstore has been added.
2017-10-23 18:07:48 +00:00
It is possible to create logical volumes on top of other devices using RPC.
2017-09-14 14:53:36 +00:00
2017-10-06 07:27:23 +00:00
See the [logical volumes ](http://www.spdk.io/doc/logical_volumes.html ) documentation for more information.
2017-09-14 14:53:36 +00:00
2017-09-26 13:21:38 +00:00
### Persistent Memory
2017-10-23 18:07:48 +00:00
A new persistent memory bdev type has been added.
The persistent memory block device is built on top of [libpmemblk ](http://pmem.io/nvml/libpmemblk/ ).
It is possible to create pmem devices on top of pmem pool files using RPC.
See the [Pmem Block Device ](http://www.spdk.io/doc/bdev.html#bdev_config_pmem ) documentation for more information.
2017-09-26 13:21:38 +00:00
2017-10-23 20:01:04 +00:00
### Virtio SCSI driver
A userspace driver for Virtio SCSI devices has been added.
The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application.
See the [Virtio SCSI ](http://www.spdk.io/doc/virtio.html ) documentation and [Getting Started ](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi ) guide for more information.
2018-01-30 21:22:20 +00:00
### Vhost target
The vhost target application now supports live migration between QEMU instances.
2017-09-26 13:21:38 +00:00
2017-07-17 23:46:33 +00:00
## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev
2017-05-17 23:28:02 +00:00
### Build System
A `configure` script has been added to simplify the build configuration process.
The existing CONFIG file and `make CONFIG_...` options are also still supported.
Run `./configure --help` for information about available configuration options.
A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk`
option is specified to configure, the SPDK build system will automatically build a
known-good configuration of DPDK with the minimal options enabled. See the Building
section of README.md for more information.
A [Vagrant ](https://www.vagrantup.com/ ) setup has been added to make it easier to
develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant
section of README.md for more information.
2017-06-30 21:26:37 +00:00
### Userspace vhost-blk target
The vhost library and example app have been updated to support the vhost-blk
protocol in addition to the existing vhost-scsi protocol.
See the [vhost documentation ](http://www.spdk.io/doc/vhost.html ) for more details.
### Block device abstraction layer (bdev)
A GPT virtual block device has been added, which automatically exposes GPT partitions
with a special SPDK-specific partition type as bdevs.
See the [GPT bdev documentation ](http://www.spdk.io/doc/bdev.md#bdev_config_gpt ) for
more information.
2017-05-17 23:28:02 +00:00
### NVMe driver
The NVMe driver has been updated to support recent Intel SSDs, including the Intel®
Optane™ SSD DC P4800X series.
A workaround has been added for devices that failed to recognize register writes
during controller reset.
The NVMe driver now allocates request tracking objects on a per-queue basis. The
number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by
modifying `io_queue_requests` in the opts structure.
The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`).
2017-07-14 23:08:05 +00:00
spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override
controller-level options for each individual I/O queue pair.
Existing callers with qprio == 0 can be updated to:
~~~
... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);
~~~
Callers that need to specify a non-default qprio should be updated to:
~~~
struct spdk_nvme_io_qpair_opts opts;
spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, & opts, sizeof(opts));
opts.qprio = SPDK_NVME_QPRIO_...;
... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, & opts, sizeof(opts));
~~~
2017-05-17 23:28:02 +00:00
### Environment Abstraction Layer
The environment abstraction layer has been updated to include several new functions
in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the
current set of functions.
2017-08-09 16:46:14 +00:00
### SPDK Performance Analysis with Intel® VTune™ Amplifier
Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018.
This analysis provides:
- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.)
- Tuning insights on the interplay of I/O and compute devices by estimating how many cores
would be reasonable to provide for SPDK to keep up with a current storage workload.
See the VTune Amplifier documentation for more information.
2017-03-28 18:31:03 +00:00
## v17.03: Blobstore and userspace vhost-scsi target
### Blobstore and BlobFS
The blobstore is a persistent, power-fail safe block allocator designed to be
used as the local storage system backing a higher-level storage service.
See the [blobstore documentation ](http://www.spdk.io/doc/blob.html ) for more details.
BlobFS adds basic filesystem functionality like filenames on top of the blobstore.
This release also includes a RocksDB Env implementation using BlobFS in place of the
kernel filesystem.
See the [BlobFS documentation ](http://www.spdk.io/doc/blobfs.html ) for more details.
### Userspace vhost-scsi target
A userspace implementation of the QEMU vhost-scsi protocol has been added.
The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices.
2017-03-29 23:01:32 +00:00
See the [vhost documentation ](http://www.spdk.io/doc/vhost.html ) for more details.
2017-02-16 00:14:02 +00:00
### Event framework
The overhead of the main reactor event loop was reduced by optimizing the number of
calls to spdk_get_ticks() per iteration.
### NVMe library
The NVMe library will now automatically split readv/writev requests with scatter-gather
lists that do not map to valid PRP lists when the NVMe controller does not natively
support SGLs.
The `identify` and `perf` NVMe examples were modified to add a consistent format for
specifying remote NVMe over Fabrics devices via the `-r` option.
This is implemented using the new `spdk_nvme_transport_id_parse()` function.
2017-01-25 23:36:40 +00:00
### iSCSI Target
The [Nvme] section of the configuration file was modified to remove the `BDF` directive
and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF)
devices can now be specified as the backing block device. A script to generate an
entire [Nvme] section based on the local NVMe devices attached was added at
`scripts/gen_nvme.sh` .
### NVMe-oF Target
The [Nvme] section of the configuration file was modified to remove the `BDF` directive
and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF)
devices can now be specified as the backing block device. A script to generate an
entire [Nvme] section based on the local NVMe devices attached was added at
`scripts/gen_nvme.sh` .
2017-02-15 23:45:58 +00:00
## v16.12: NVMe over Fabrics host, hotplug, and multi-process
2016-12-07 22:47:01 +00:00
### NVMe library
2016-10-04 17:23:07 +00:00
The NVMe library has been changed to create its own request memory pool rather than
requiring the user to initialize the global `request_mempool` variable. Apps can be
updated by simply removing the initialization of `request_mempool` . Since the NVMe
library user no longer needs to know the size of the internal NVMe request
structure to create the pool, the `spdk_nvme_request_size()` function was also removed.
2016-10-04 23:01:34 +00:00
The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become
`spdk_nvme_ns_cmd_dataset_management()` , which allows access to all of the NVMe
Dataset Management command's parameters. Existing callers can be updated to use
`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the
`type` parameter.
2016-11-03 17:12:16 +00:00
The NVMe library SGL callback prototype has been changed to return virtual addresses
rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and
`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match.
2016-12-07 22:47:01 +00:00
The NVMe library now supports NVMe over Fabrics devices in addition to the existing
support for local PCIe-attached NVMe devices. For an example of how to enable
NVMe over Fabrics support in an application, see `examples/nvme/identify` and
`examples/nvme/perf` .
Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug
support, an application should call the `spdk_nvme_probe()` function on a regular
basis to probe for new devices (reported via the existing `probe_cb` callback) and
removed devices (reported via a new `remove_cb` callback). Hotplug is currently
only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe
devices must be bound to `uio_pci_generic` by an external script or tool.
2016-12-13 00:26:58 +00:00
Multiple processes may now coordinate and use a single NVMe device simultaneously
using [DPDK Multi-process Support ](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html ).
2016-12-07 22:47:01 +00:00
### NVMe over Fabrics target (`nvmf_tgt`)
The `nvmf_tgt` configuration file format has been updated significantly to enable
new features. See the example configuration file `etc/spdk/nvmf.conf.in` for
more details on the new and changed options.
The NVMe over Fabrics target now supports virtual mode subsystems, which allow the
user to export devices from the SPDK block device abstraction layer as NVMe over
Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported,
and a single `nvmf_tgt` may export both types of subsystems simultaneously.
2016-12-13 00:26:58 +00:00
### Block device abstraction layer (bdev)
The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe
blockdev driver has been updated to support scatter/gather. Apps can use the
new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()`
functions.
The bdev status returned from each I/O has been extended to pass through NVMe
or SCSI status codes directly in cases where the underlying device can provide
a more specific status code.
A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the
`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or
NVMe namespaces.
2016-12-07 22:47:01 +00:00
### General changes
`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is
2016-08-08 22:57:49 +00:00
used instead. Prior to DPDK 16.07 enumeration by class code was not supported,
2016-12-07 22:47:01 +00:00
so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered
by the NVMe library.
The `env` environment abstraction library has been introduced, and a default
DPDK-based implementation is provided as part of SPDK. The goal of the `env`
layer is to enable use of alternate user-mode memory allocation and PCI access
2017-01-17 23:23:28 +00:00
libraries. See `doc/porting.md` for more details.
2016-08-08 22:57:49 +00:00
2016-11-17 20:47:43 +00:00
The build process has been modified to produce all of the library files in the
`build/lib` directory. This is intended to simplify the use of SPDK from external
projects, which can now link to SPDK libraries by adding the `build/lib` directory
to the library path via `-L` and linking the SPDK libraries by name (for example,
`-lspdk_nvme -lspdk_log -lspdk_util` ).
2016-12-07 22:47:01 +00:00
`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user
to query and modify the configuration at runtime. The RPC service is disabled by
default, since it currently does not provide any authentication or security
mechanisms; it should only be enabled on systems with controlled user access
behind a firewall. An example RPC client implemented in Python is provided in
`scripts/rpc.py` .
2017-02-15 23:45:58 +00:00
## v16.08: iSCSI target, NVMe over Fabrics maturity
2016-08-08 23:35:11 +00:00
2016-08-04 20:28:25 +00:00
This release adds a userspace iSCSI target. The iSCSI target is capable of exporting
NVMe devices over a network using the iSCSI protocol. The application is located
in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in.
2016-06-28 22:27:50 +00:00
2016-08-04 20:28:25 +00:00
This release also significantly improves the existing NVMe over Fabrics target.
2016-06-28 22:27:50 +00:00
- The configuration file format was changed, which will require updates to
any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in` ):
- `SubsystemGroup` was renamed to `Subsystem` .
- `AuthFile` was removed (it was unimplemented).
- `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names)
when naming subsystems. The default node name was changed to reflect this;
it is now "nqn.2016-06.io.spdk".
2016-08-04 20:28:25 +00:00
- `Port` and `Host` sections were merged into the `Subsystem` section
- Global options to control max queue depth, number of queues, max I/O
size, and max in-capsule data size were added.
- The Nvme section was removed. Now a list of devices is specified by
bus/device/function directly in the Subsystem section.
- Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt
to future-proof the interface, so the only mode supported by this release
is "Direct".
2016-06-28 22:27:50 +00:00
- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library.
2016-08-04 20:28:25 +00:00
- The target now supports discovery.
This release also adds one new feature and provides some better examples and tools
for the NVMe driver.
- The Weighted Round Robin arbitration method is now supported. This allows
the user to specify different priorities on a per-I/O-queue basis. To
enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()` .
- A simplified "Hello World" example was added to show the proper way to use
the NVMe library API; see `examples/nvme/hello_world/hello_world.c` .
- A test for measuring software overhead was added. See `test/lib/nvme/overhead` .
2016-06-28 22:27:50 +00:00
2017-02-15 23:45:58 +00:00
## v16.06: NVMf userspace target
2016-06-06 23:40:34 +00:00
This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the
newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe
devices from a host machine over the network via RDMA. Currently, the target is
limited to directly exporting physical NVMe devices, and the discovery subsystem
is not supported.
2016-03-08 20:34:24 +00:00
This release includes a general API cleanup, including renaming all declarations
in public headers to include a `spdk` prefix to prevent namespace clashes with
user code.
- NVMe
- The `nvme_attach()` API was reworked into a new probe/attach model, which
moves device detection into the NVMe library. The new API also allows
parallel initialization of NVMe controllers, providing a major reduction in
startup time when using multiple controllers.
2016-02-29 21:19:02 +00:00
- I/O queue allocation was changed to be explicit in the API. Each function
that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`)
argument, and I/O queues may be allocated using
`spdk_nvme_ctrlr_alloc_io_qpair()` . This allows more flexible assignment of
queue pairs than the previous model, which only allowed a single queue
per thread and limited the total number of I/O queues to the lowest number
supported on any attached controller.
2016-03-08 20:34:24 +00:00
- Added support for the Write Zeroes command.
- `examples/nvme/perf` can now report I/O command latency from the
the controller's viewpoint using the Intel vendor-specific read/write latency
log page.
- Added namespace reservation command support, which can be used to coordinate
sharing of a namespace between multiple hosts.
- Added hardware SGL support, which enables use of scattered buffers that
don't conform to the PRP list alignment and length requirements on supported
NVMe controllers.
2016-04-28 22:59:17 +00:00
- Added end-to-end data protection support, including the ability to write and
read metadata in extended LBA (metadata appended to each block of data in the
buffer) and separate metadata buffer modes.
See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()`
for details.
2016-03-08 20:34:24 +00:00
- IOAT
- The DMA block fill feature is now exposed via the `ioat_submit_fill()`
function. This is functionally similar to `memset()` , except the memory is
filled with an 8-byte repeating pattern instead of a single byte like memset.
- PCI
- Added support for using DPDK for PCI device mapping in addition to the
2016-04-28 22:59:17 +00:00
existing libpciaccess option. Using the DPDK PCI support also allows use of
the Linux VFIO driver model, which means that SPDK userspace drivers will work
with the IOMMU enabled. Additionally, SPDK applications may be run as an
unprivileged user with access restricted to a specific set of PCIe devices.
2016-03-08 20:34:24 +00:00
- The PCI library API was made more generic to abstract away differences
between the underlying PCI access implementations.
2017-02-15 23:45:58 +00:00
## v1.2.0: IOAT user-space driver
2016-03-08 20:34:24 +00:00
This release adds a user-space driver with support for the Intel I/O Acceleration Technology (I/OAT, also known as "Crystal Beach") DMA offload engine.
- IOAT
- New user-space driver supporting DMA memory copy offload
- Example programs `ioat/perf` and `ioat/verify`
- Kernel-mode DMA engine test driver `kperf` for performance comparison
- NVMe
- Per-I/O flags for Force Unit Access (FUA) and Limited Retry
- Public API for retrieving log pages
- Reservation register/acquire/release/report command support
- Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks
- Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features
- Updated to support DPDK 2.2.0
2017-02-15 23:45:58 +00:00
## v1.0.0: NVMe user-space driver
2016-03-08 20:34:24 +00:00
This is the initial open source release of the Storage Performance Development Kit (SPDK).
Features:
- NVMe user-space driver
- NVMe example programs
- `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver
- `examples/nvme/identify` displays NVMe controller information in a human-readable format
- Linux and FreeBSD support
