These need to be available for the lifetime of the probe_info structure,
so they can't be pointing at e.g. temporary buffers on the stack.
Change-Id: I5aaa898acf9314aab51600dd756f966965d37fd0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It always points to the same internal RDMA request complete function, so
just call that function directly.
Change-Id: Ic1fb6236bf43eaad62413df77d43be9ab855e5c7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
We can't transfer more than the bounce buffer in a single command, so
report that rather than some bogus value.
Change-Id: I39b147916dcc2ee478470917298763a239a6a35a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Record the user-provided asynchronous event configuration set via Set
Features, and return it in Get Features.
This value is not actually used, since AER is not implemented yet in the
virtual controller model, but it at least implements the mandatory
Set/Get Features.
This allows the hack in the NVMe host code that ignored the Set Features
failure to be reverted.
Change-Id: I2ac639eb8b069ef8e87230a21fa77225f32aedde
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fill in the cached copy of CAP in the generic NVMe controller to match
the PCIe transport.
This is not really early enough, since CAP is used during the reset
process to determine the reset timeout, but that will have to be fixed
separately by rearranging some of the transport callbacks.
Change-Id: Ia8e20dbb8f21c2871afb9e00db56d0730e597331
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make sure the entire NQN field is zero-padded, rather than using
strlen() on the input.
Change-Id: Icee68bd033feed057813beeb30cec102ed90840e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This fixes a compiler warning about unhandled enum cases in a switch.
Change-Id: Icecb56b47a05c13f390f03b877f8eae243b481a6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
- add SPDK_NVME_OPC_KEEP_ALIVE to admin_opcode
- add SPDK_NVME_SC_INVALID_SGL_OFFSET, SPDK_NVME_SC_INVALID_SGL_OFFSET,
SPDK_NVME_SC_HOSTID_INCONSISTENT_FORMAT, SPDK_NVME_SC_KEEP_ALIVE_EXPIRED
and SPDK_NVME_SC_KEEP_ALIVE_INVALID to generic_status
The SPDK_TRACELOG macro depends on a CONFIG setting (DEBUG), so it
should not be part of the public API.
Create a new include/spdk_internal directory for headers that should
only be used within SPDK, not exported for public use.
Change-Id: I39b90ce57da3270e735ba32210c4b3a3468c460b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Considering the process can be terminated in the cases like ctrl+c,
kill command or memory fault, the ref is tracked in the per process
structure spdk_nvme_controller_process and whenever there is other
process attaches or detaches the controller, a scan will be issued
to cleanup those unexpectedly exited processes.
Change-Id: Ib4f974f567a865748d42da4ead49edd383dfc752
Signed-off-by: GangCao <gang.cao@intel.com>
Instead of the next_sge callback returning the physical address
directly, make it return the virtual address and convert to physical
address inside the NVMe library.
This is necessary for NVMe over Fabrics host support, since the RDMA
userspace API requires virtual addresses rather than physical addresses.
It is also more consistent with the normal non-SGL NVMe functions that
already take virtual addresses.
Change-Id: I79a7af64ead987535f6bf3057b2b22aef3171c5b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Function pointers will not work for the DPDK multi-process model (they
can have different addresses in different processes), so define a
transport enum and dispatch functions that switch on the transport type
instead.
Change-Id: Ic16866786eba5e523ce533e56e7a5c92672eb2a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The initialization of dev_addr was replaced with probe_info.pci_addr,
but its use in spdk_pci_addr_compare() wasn't replaced to match.
Fixes commit fcb00f3780 (nvme: expand
probe information to a struct).
Change-Id: Ic4c273d2aa0bf1f9e3e1527f3ab09d3c019158cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
spdk_nvme_probe() will now provide a struct spdk_nvme_probe_info to the
probe and attach callbacks in place of the PCI device pointer.
This struct contains the useful information that could be retrieved from
the PCI device during probe.
The goal of this change is to allow expansion of the probe information
in the future when other transports (specifically, NVMe over Fabrics)
are added that do not necessarily use PCI addressing or device IDs.
Change-Id: I59a2a9e874e248ce5fa1d7f4b57c8056962ff3cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the new public PCI ID structure in the NVMe library to replace the
previously private struct pci_id.
Change-Id: I267d343917f60bdae949a824bc0fe67457cbbc0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
- Split the part that gets a PCI device's address into its own function,
spdk_pci_device_get_addr(). This is useful outside of the comparison
function and is orthogonal to comparing addresses.
- Make the comparison function take two addresses instead of a device
and an address. The more general form will be useful with addresses
that are not directly associated with a device. Because of this, also
rename the function from spdk_pci_device_compare_addr() to
spdk_pci_addr_compare().
- Return a signed value similar to strcmp() so that addresses can be
ordered, not just compared for equality.
Change-Id: Idf304454af09ea57f1e1d5dc3a39b077378cecad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a field to struct spdk_nvme_ctrlr_opts that allows the user to
specify a keep alive timeout, and add automatic submission of Keep Alive
commands to spdk_nvme_ctrlr_process_admin_completions().
Change-Id: Ib282299a571d8edc59c7933418751bc3a6c98b40
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the quirks mechanism generic in preparation for quirks for devices
from other vendors.
Change-Id: Ic003b020a38f1b966021db30e3f2bce9cf6a1a0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add a transport function to get the max data transfer size to break the
dependency on NVME_MAX_XFER_SIZE.
Change-Id: I846d12878bdd8b80903ca1b1b49b3bb8e2be98bb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Move the PCIe-specific admin queue setup to nvme_pcie_ctrlr_enable.
Change-Id: Ic3f5625fa804f719040ba86b7fc3bf82fcc057c0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The value of CAP should not change during the lifetime of a controller,
so read it once during ctrlr_construct and store it in the ctrlr.
Change-Id: I089d4141b4e0c9aae6c53abf9bb0ef6577dabe0b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than embedding adminq directly in the spdk_nvme_ctrlr structure,
change it to a pointer to a spdk_nvme_qpair. This is necessary to allow
the transport to extend the qpair structure.
Change-Id: I041685d5037088cf56d046fe99bf204edcfc57b1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This requires a couple of related changes:
- I/O queue IDs are now allocated by using a bit array of free queue IDs
instead of keeping an array of pre-initialized qpair structures.
- The "create I/O qpair" function has been split into two: one to create
the queue pair at startup, and one to reinitialize an existing qpair
structure after a reset.
Change-Id: I4ff3bf79b40130044428516f233b07c839d1b548
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the transport ctrlr_construct callback responsible for allocating
its own controller.
Change-Id: I5102ee233df23e27349410ed063cde8bfdce4c67
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These are specific to local NVMe PCIe devices, so move them out of the
generic NVMe code into the PCIe transport.
Change-Id: Iea2056a4c438b7d3a303b4b5e977ce7aa9e58c05
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow factoring out PCIe-specific code into a swappable
transport so that NVMe over Fabrics host support can be added.
Change-Id: I4df74dd268d655e3b36e8d6114ebe7d79a24844d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For the nvme readv/writev APIs, the PRP checking logic was
incorrectly failing single SGE payloads that were larger
than 4KB. This patch adds a test case for this scenario,
and fixes the PRP checking logic.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6357d620599666046d2cb74d7923dac1f75418c5
Enforce exactly one trailing \n, and fix all of the existing cases.
Change-Id: I6218e4700e90aeb647eaee78089530c79993c8c8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows users to swap their PCI library from
libpciaccess/dpdk to another mechanism using the standard
method for swapping out the env library.
Change-Id: Ib2248f8b43754a540de2ec01897e571f0302b667
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This allows users to swap out SPDK's third party
libraries for an implementation based on their own
framework.
Change-Id: Ia0b7384ce5e31acba5ad0d7002dec9e95b759c52
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The new env library will wrap all third-party library
calls and be easily swappable with alternate implementations
at build time. For now, it's just the memory library
renamed.
Change-Id: I26a70933289f8137107208ba75f7520fd7a33da0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This patch also drops support for automatically unbinding
devices from the kernel - run scripts/setup.sh first.
Our generic pci interface is now hidden behind include/spdk/pci.h
and implemented in lib/util/pci.c. We no longer wrap the calls
in nvme_impl.h or ioat_impl.h. The implementation now only uses
DPDK and the libpciaccess dependency has been removed. If using
a version of DPDK earlier than 16.07, enumerating devices
by class code isn't available and only Intel SSDs will be
discovered. DPDK 16.07 adds enumeration by class code and all
NVMe devices will be correctly discovered.
Change-Id: I0e8bac36b5ca57df604a2b310c47342c67dc9f3c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Provide a convenience wrapper for general purpose dataset
management commands. The previous wrapper for deallocate
was difficult to use correctly and only for deallocate.
Note that the name is "dataset_management" as opposed to
"data_set_management" to match the NVMe specification.
It's questionable whether "dataset" is valid English, but
it is best to match the specification.
Change-Id: Ifc03d66dbabeabe8146968cf8a09f7ac3446ad68
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Namespaces can be allocated but inactive, which causes
the identify namespace command to fail. Handle this
case so that attaching to the controller does not fail.
Change-Id: I9d692f8e7841a9315a737b0a5e44d9b4e4484a13
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Rather than forcing the NVMe library user to pass a specially-allocated
block of memory (e.g. rte_malloc() in the case of the default
nvme_impl.h), just make the NVMe library allocate a suitable buffer
itself and copy to/from the user buffer as needed.
The fast path I/O functions still require special rte_malloc()
allocations, since we don't want to add an allocation and copy to the
I/O critical path.
Change-Id: I7fe88c0ba60c859a33bbe95b7713f423c6bf1ea8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Intel DC P3*** NVMe devices specify a desired stripe size, which was
used for splitting I/O. Not all devices, however, specify a desired
stripe size (such as the Intel DC D3*** line), and for only these
devices there was a logic mistake that overwrote the maximum I/O
size with a 2MB default. This patch corrects that error.
Change-Id: I94b72a3a3dd1dfa18bd638daf7e01a592eb6ed17
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
DPDK 16.07 introduced a new PCI ID field for matching by class code
instead of vendor/device ID. Use it to match all NVMe devices instead of
explicitly listing vendor and device ID pairs.
Change-Id: Ib2a5cc6833bf2b793d37d77caab97207f365df8f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
assert is part of the C standard library and is available
on any platform we'd consider porting to. Don't put a
wrapper around it.
Change-Id: I0acfdd6a8a269d6c37df38fb7ddf4f1227630223
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
pthreads are widely supported and are available on any
platform we currently foresee porting to. Use that API
instead of attempting to abstract it away to simplify
the code.
Change-Id: I822f9c10910020719e94cce6fca4e1600a2d9f2a
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Since we bind the NVMe device to UIO driver to protect against native
NVMe driver, but for Admin queue, there are still INTx interrupts
exist, as all the completion for Admin queue will be processed in
user space, so we don't need INTx anymore.
Change-Id: Ife5b3e410ae95690ed0f3f9a2f2dfaf55a7797b5
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
rte_zmalloc() is broken and does not actually return zeroed memory on at
least DPDK 16.07 on FreeBSD, so do it ourselves.
Change-Id: If8da93ead0b3911c8bca24aa27ed90dc00b8a9a4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is a step towards enabling sharing SPDK NVMe
device access from multiple processes using DPDK's
multi-process framework.
Change-Id: I57d5eec158b42addc1036bd2583596471a467a95
Signed-off-by: GangCao <gang.cao@intel.com>
Drop the special-case preprocessor definition for PCI access library now
that config.h is available with an equivalent SPDK_CONFIG_PCIACCESS
define.
Change-Id: I4891d0f2fd7d3eea51b767df9e594555b36265ea
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Explicitly include spdk.common.mk at the top of all lib Makefiles so
that CONFIG options and other predefined variables are set.
Change-Id: I1e560c294fe8242602e45191a280f4295533ae44
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For a single poll of the completion queue, if the user
submits I/O from within their completion callback and their
completion callback is particularly slow to execute, the loop
could potentially continue forever. To support this, we
need to limit the number of completions we'll process
in one batch.
Change-Id: If6bae47e52b36347dbe5622ace68c866ee88a0b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This is intended to be used for examples/nvme/identify and similar
diagnostic utilities.
Change-Id: Ib2f941e9af7a3fb7555865ef253742e30ccad2b5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Previously, we used cap_lo and cap_hi to represent the 32-bit halves of
the full CAP register. However, it is simpler to keep them in a single
64-bit structure, and is no less efficient on 64-bit platforms.
Also name the NSSRS field from NVMe 1.2, which was previously reserved.
Change-Id: I1d5d9b0dccbb12373b4aed3db29c883881d43223
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will allow removal notifications to be propagated to the library
user (e.g. for hotplug).
The callback is currently unused, but this at least prepares the API for
the future hotplug support.
Based on a patch by Dave Jiang <dave.jiang@intel.com>
Change-Id: I20b1c2dbf5e084e0b45a7e51205aba4514ee9a95
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use the knowledge that both the source and destination of
nvme_copy_command() are aligned to emit the aligned variants of the
SSE2/AVX mov instructions.
Change-Id: I0a7e32a3bb10b9a1920cd85691b79fa7172eecb3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This moves some definitions from nvmf_spec.h to
nvme_spec.h based on the latest publication.
Change-Id: I51b0abd16f7d034696239894aea5089f8ac70c40
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Swap the order of checks in the failure check - if rc is not 0, addr may
be garbage.
Change-Id: I110710efd00397c777d59ac8b219ba3cc2156596
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The conventional rule for returning errno is negative, hence there is no
need to modify caller's code to adjust this NVMe library.
Signed-off-by: Minfei Huang <mnghuan@gmail.com>
Signed-off-by: Minfei Huang <minfei.hmf@alibaba-inc.com>
Previously, there was no way to determine what namespace ID was assigned
when a namespace was created via the NVMe library interface.
Also drop the incorrect comment about calling
spdk_nvme_ctrlr_process_admin_completions(), since
spdk_nvme_ctrlr_create_ns() checks the admin queue internally.
Change-Id: If90a6e9fc773aefa220ebbf6effc2d033c9f20cc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This lets spdk_nvme_qpair fit in 128 bytes exactly.
Change-Id: I7c42582f22ece72a7f1d651468e63d4fe05babd6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Avoid potential double free cases.
This fixes a clang warning during scan-build.
Change-Id: I487d6fcd485d1f8ebb96b6f8cb54511628461f39
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
All trackers in outstanding_tr should have a non-NULL request. Add an
assert to verify this.
Fixes a clang warning during scan-build.
Change-Id: I0ac4d2bad17449f684808cbb98777627d890b65b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Document the requirement that qpair and cmd can't be NULL.
This placates clang, which previously generated a warning
during scan-build.
Change-Id: Ic2d5e808faee0028c890ce1312444fb3dc95f223
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The user can determine whether submission queues will be placed in the
controller memory buffer by checking the controller options use_cmb_sqs
flag in the attach callback.
Change-Id: I8a925ef99a48665a0e2ffaa90d9ff2b79b90b2fa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The D3700/D3600 series support Controller Memory Buffer(CMB) feature,
CMB is available for holding submission queues, for those controllers
which can support submission queues in CMB, user can set the option
whether to enable it or not.
Change-Id: I8b0dc9e28dd6f5bb01bee99a532087212c04e492
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Resolve relative paths before using them to clean up command lines.
This should also help shorten the overall command line length that gets
embedded in the binary and used when locating the executable from a
coredump.
Change-Id: Ibff9849ede198bb04313496c8b7131485ffaf14f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch add support for Intel specific log pages :
marketing description page.
Change-Id: I87bccb2af286279598c9dd3c870094b384a0d2f7
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
For existing \file markers, move them to the top of the header and tweak
the wording for consistency.
Change-Id: Icce748effe4dbe97d79a8c87d31caf0ee5797058
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For those controllers which can support end-to-end data protection
feature, add the support in the driver layer.
Change-Id: Ifac3dd89dec9860773c850416a6116113a6ce22a
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
When using DPDK PCI support via VFIO, the PCI device is reset
immediately before calling the PCI driver's init function. In some
cases, the device seems to not be ready to handle MMIO accesses right
away. Until the cause of this issue is fully understood, add a 500 ms
sleep as a workaround.
Change-Id: Ic893080a6f34d57eee80df3e6aa68c220c08df3e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
All children must be removed from a parent request before the parent is
freed.
Change-Id: I073ff0e9c5bcdd6181d90b918bfe4cce054f6c0b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename nvme_remove_child_request() to nvme_request_remove_child() and
move it next to nvme_request_add_child() to make the symmetry clear.
Change-Id: I78747c44ab3db1a656b33555a45f634dc5a55b31
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The NVMe specification recommends destroying all I/O submission and
completion queues before setting CC.SHN.
Change-Id: Iad71dd3fe03d897858034f3ca6ee02e0c55cc2b0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The NVMe specification recommends that orderly shutdown should just
write CC.SHN while the controller is still enabled rather than writing
CC.EN = 0 first.
This also allows removal of the now-unused nvme_ctrlr_disable() and
nvme_ctrlr_wait_for_ready() functions.
Change-Id: I4702ffda153f218ebb8ed92f0e36144b7ceded93
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This can happen if the controller is still resetting as the SPDK NVMe
driver takes control.
Change-Id: I263ae8f2e7b271e0448450557452a115c90c4fb6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch is used to handle the memory leak issue when
a parent nvme_request is free. In our current code,
we did not free the nvme_request allocated by the children
in the exceptional case.
Change-Id: Iabd1f1c3594af60c38e74e3d96c14f78d1aa1aed
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
This patch is used to add a nvme_request remove child
helpler function
Change-Id: I1e5bb228d53333ca3601f4ae30fcd801ea39e532
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Drop the "data_transfered" variable and just update length, since length
was not used otherwise after this point in the loop.
Change-Id: Icd2991e4e85de7e8c951ba14c441434e871ea4ef
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If only one descriptor is needed, there is a special case in the spec for
SGL1 using the Data Block descriptor type. Add a comment to make it
clear what is going on.
Also tweak the SGL1 setup to copy from the first SGL descriptor list
element instead of relying on the last value from the loop above, since
that could be easily broken by accident.
Change-Id: I49ef97fe5bf18d2bf1d86b4310a7d3abdfd03e57
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
tr->u.sgl is already a struct spdk_nvme_sgl_descriptor pointer.
Change-Id: Ie2c8c052fc28e6369d1d095b8d566acae47975d1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
_nvme_qpair_build_hw_sgl_request() will only be called for payload_size
!= 0, so every SGL will have at least one segment. Drop the 'else' that
was handling nseg == 0, and add an assert to document the payload_size
requirement.
Change-Id: I48e2a862a7657ba85605c0d35c0b65dfac072167
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The assert is checking a variable that is invariant within the loop, so
move the assert up to the top of the function.
Change-Id: Iee7eea1736bc7f953665feb390c3d6340dbeffbc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This makes it easier to find the larger doc comments that produce separate
pages.
It also allows removing the lib/nvme directory from the Doxyfile, so
only the public API headers are used to generate documentation.
Change-Id: I8c46edb8067a91dda5b23fb0864efd3dd8aaeba5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
spdk_nvme_qpair_process_completions() is already documented in
spdk/nvme.h, so merge the doc comment from nvme_qpair.c into the public
header.
Change-Id: Id7722d99d209852ee64286e0a3fa127b863e10aa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Don't allow the user to request more than the valid maximum number of
I/O queues (65535) or 0 I/O queues, since this can't be encoded.
Change-Id: I2d6e0bba03476085842bad683b273cdf9d6e6d5e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Per the NVMe spec, SGL segments must be Qword (8-byte) aligned. Add a
static assert to make sure this is true for the sgl member of struct
nvme_tracker (assuming the whole nvme_tracker is at least 8-byte aligned).
Change-Id: I827aa40b56de648d83f524a4f1e79c3202b676be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If the controller is failed, attempting to submit additional I/O is
futile - it will be immediately failed using the completion callback,
which can result in infinite recursion if the application code resubmits
I/Os on failure.
Instead, provide a way for request submission to indicate failure, and
use it to exit early if the controller is failed; this can only happen
when a reset failed (timed out).
If a request is submitted directly by the user when the controller has
failed, we can return an error code directly. For the case where I/O
was queued and is being resubmitted after a reset, we still need to call
the completion handler via _nvme_fail_request_ctrlr_failed().
Change-Id: I9e144328d524b25db2acf48e923b584746e8d0b6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Provide a new structure, spdk_nvme_ctrlr_opts, to let the user modify
the default controller initialization options during probe/attach.
Currently, only the number of queue pairs can be modified in this way;
other options will be added later.
Change-Id: Ie27b9429291d93a9353c0d820f0ad467d3b0e7cb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Replace the previous code that allocated each tracker individually with
one large allocation per queue pair.
struct nvme_tracker is now explicitly padded to reach exactly 4096 bytes
to allow normal array indexing to work correctly while maintaining the
alignment requirement that ensures each tracker's PRP list does not
cross a page boundary.
This also allows removal of the act_tr array, since the tr array can be
indexed directly now, and each tracker can store its own active state.
Change-Id: Ia7c51735b96594d12f7f478cefcc4aedc84207ad
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The previous method for registering I/O queues did not allow the user
to specify queue priority for weighted round robin arbitration, and it
limited the application to one queue per controller per thread.
Change the API to require explicit allocation of each queue for each
controller using the new function spdk_nvme_ctrlr_alloc_io_qpair().
Each function that submits a command on an I/O queue now takes an
explicit qpair parameter rather than implicitly using the thread-local
queue.
This also allows the application to allocate different numbers of
threads per controller; previously, the number of queues was capped at
the smallest value supported by any attached controller.
Weighted round robin arbitration is not supported yet; additional
changes to the controller startup process are required to enable
alternate arbitration methods.
Change-Id: Ia33be1050a6953bc5a3cca9284aefcd95b01116e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Since the nvme_tracker struct was extended to allow space for 253 SGL
descriptors at 16 bytes each, we can use the same amount of space in the
other branch of the union to store 506 PRP list entries at 8 bytes each.
This increases the maximum supported I/O size for PRP-only devices from
128 KB to slightly under 2 MB.
Change-Id: I2b9905be41343ff360b4cdaccca87ea6f753e89c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Also add a compile-time assert to make sure this doesn't accidentally
break again in the future.
Change-Id: I4d18cfbf21392291e1bdd76eff055429009d28d6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_ctrlr_cmd_create_io_cq() was using the queue ID as the
IV (Interrupt Vector) field in the Create I/O Completion Queue command.
Since the SPDK NVMe driver does not enable interrupts, this is
misleading at best.
Change-Id: I3ea53701fdb9f21d9dc8d8fe20ccf2833b76cfbf
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch just implement the feature of format progress indicator.
the NVMe available does't support FPI currently.
Change-Id: Ie937591fb1720d8a062354322aabcc95ff14b2d3
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
This field is write-only in the current code; the NVMe library does
not track timeouts on requests.
Change-Id: I50e53bb3c299bf16912c48be8aad3eec829154af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
When I/O spans a stripe boundary, the driver splits the request into
multiple requests, so for 1 segment memory larger than the stripe
size, we also need to split the segment memory.
Change-Id: I22ea5734d7066865a57a3c90fe18d5f76f373f1d
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
For those NVMe controllers which can support SGL feature in
firmware, we will use SGL for scattered payloads.
Change-Id: If688e6494ed62e8cba1d55fc6372c6e162cc09c3
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
nvme_spec.h already has a structure with the correct bitfields for the
CSTS register, so use it in struct spdk_nvme_registers.
Change-Id: Id0663aee2611fb5195f9012a3176799e32701bb0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This will be exposed in the public API. This rename is in a separate
commit to ease review.
Change-Id: I1b7fef36f85265db27935ac4d22ceef3c7282502
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Set up the infrastructure for creating I/O submission queues with
variable queue priority (QPRIO in Create I/O SQ command).
Currently, this is unused, since we always use the default arbitration
method (round robin), but it will allow reinitializing submission queues
with the correct priority once weighted round robin is supported.
Change-Id: I425003879e624cfcc9687bdc495b5c1726b5a8af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This function no longer exists and was not part of the public API.
Change-Id: I94fd066b63e812367687d11bc00aa11ab88d4671
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Many of the internal controller initialization functions did not check
for allocation failure; add return codes and check them where
applicable.
Change-Id: Id1b33bb06fca84035369d8b7ecd4c36b8ba7134c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This function returns true if the namespace is active or false if it is
inactive (e.g. no namespace has been attached to the specified namespace
ID yet).
Also use the new function to add checks in the examples and tests where
applicable.
Change-Id: I35465b315ae1a1677c5a82191ad9b1da1c216d50
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Prepare for qpair to be exposed as part of the public API.
Change-Id: Ia63e863e95554adceeade20c829f12fe346375d5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
When multiple NVMe controllers are being initialized during
spdk_nvme_probe(), we can overlap the hardware resets of all controllers
to improve startup time.
Rewrite the initialization sequence as a polling function,
nvme_ctrlr_process_init(), that maintains a per-controller state machine
to determine which initialization step is underway. Each step also has
a timeout to ensure the process will terminate if the hardware is hung.
Currently, only the hardware reset (toggling of CC.EN and waiting for
CSTS.RDY) is done in parallel; the rest of initialization is done
sequentially in nvme_ctrlr_start() as before. These steps could also be
parallelized in a similar framework if measurements indicate that they
take a significant amount of time.
Change-Id: I02ce5863f1b5c13ad65ccd8be571085528d98bd5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This check was dead code, since both places that called
nvme_ctrlr_wait_for_ready() could only ever have cc.en = 1.
Remove the original nvme_ctrlr_wait_for_ready() wrapper and rename
_nvme_ctrlr_wait_for_ready() without the underscore to replace it.
Change-Id: I6c9aa6a5b93606fb89d168c23f6735fcf3a84eaa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
In nvme_ctrlr_hw_reset(), if we encounter a controller whose CC.EN bit
is already 0 (controller is disabled), the previous code would enable
the controller just so that it could be disabled to get a full reset
(transition from CC.EN = 1 to CC.EN = 0). However, it is a safe
assumption that if CC.EN is already 0, the controller has just been
reset, so we don't need to reset it again.
This saves a significant amount of time (2+ seconds per controller with
Intel SSD DC P3700) during initialization for devices that were disabled
on startup.
Change-Id: I552b1f0f185a84a8a0ce57a93b012d9d5fe096f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Explicitly include system headers for types that are used in public
headers.
These were being pulled in by example code, so SPDK itself would build,
but other apps that did not include stdbool.h would fail to compile when
including spdk/nvme.h.
Also include nvme.h first in nvme_internal.h so this case gets tested
during normal compilation.
Change-Id: I8ed0fc4e0dcf71551738c461b4b825cc2ee1d233
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
payload here is a pointer to the buffer, not a struct nvme_payload.
Use nvme_allocate_request_contig() and pass the length in bytes rather
than dwords.
Change-Id: Idbbb3614b1d69148fe041d26e0c148bd9ce53724
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The parent field is in the cache line of nvme_request that is only
supposed to be accessed for split (child) I/Os. All accesses to parent
are done from child-specific calls now, so it does not need to be
initialized in the common case of a non-split I/O.
nvme_request_add_child() will set parent when splitting occurs.
Change-Id: Ib86c16ba1ea2ce32f62079831101da2a099047af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Allows simplification of _nvme_qpair_build_sgl_request(), which does not
need to know whether a request is a child or not.
This also removes a read of req->parent for non-split I/Os; the parent
field is in the section of nvme_request that is only intended to be
initialized for split I/Os, which should be detected by looking at
num_children.
Additionally, this fixes a potential problem if requests were nested
more than one level deep (e.g. req->parent was not the original user
request).
Change-Id: I3ea1dc134bbb1e3b8c6b5a479f5d760bd97ea848
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Building the tip of the spdk master against the dpdk-2.2.0 fails with
inappropriate RTE_CACHE_LINE_SIZE error. The simple reversal of the RTE
include file order below fixed it for me.
Change-Id: I8782b7ee21d7f185e6e678f874fbdab9403117a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch is used to remove the duplicated code.
As we found the structure in "ifdef and else" are same.
Change-Id: I1717ce3dcc14134ac59c165d801e5e811b987be5
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Move toward collecting PCI IDs, class codes, etc. in pci_ids.h instead
of individual device-specific headers.
Change-Id: Icff162d48ac663db71d0576ceee16a9bd7a751cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
PCI_VENDOR_ID_INTEL -> SPDK_PCI_VID_INTEL
Also change the inclusion guard macro to be consistent with the other
SPDK headers.
Change-Id: I29346267172cb8c07cc4289eed4eca2d55e942d6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This doesn't need to be part of the public API. It is only used by the
NVMe quirk lookup tables.
Change-Id: I7662e333c70b7c5f814bd7c8a528b6bff1f0732e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename all functions with a spdk_ prefix, and provide enough of an API
to avoid apps needing to #include <pciaccess.h>.
The opaque type used in the public API for a PCI device is now
struct spdk_pci_device *.
Change-Id: I1e7a09bbc5328c624bec8cf5c8a69ab0ea8e8254
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is a step toward abstracting PCI access so that libpciaccess can be
swapped out more easily.
Change-Id: I5491459460cbfbd0be471f70f9d07a7eb3175234
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Instead of writing the completion doorbell once per completion,
just write it once at the end of the completion while loop.
This reduces the number of mmio writes by coalescing several
writes into one when we get multiple completions at a time.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3cc3864dcfe43186bec51be1a732e84ef3be05ae
This patch did the work to use pci related functions
provided by DPDK.
Change-Id: I263b79f1b42868ef0c1efcf1bc392a4b3a328e93
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
With CONFIG_PCIACCESS=y in CONFIG file, we can
use libpciaccess library; With CONFIG_PCIACCESS=n
in CONFIG file, we use pciaccess functions provided
in DPDK.
Change-Id: I786c5589b8e7909ba2e59d222938dd5ba45bf92d
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The new probing API will find all NVMe devices on the system and ask the
caller whether to attach to each one. The caller will then receive a
callback once each controller has finished initializing and has been
attached to the driver.
This will enable cleanup of the PCI abstraction layer (enabling us to
use DPDK PCI functionality) as well as allowing future work on parallel
NVMe controller startup and PCIe hotplug support.
Change-Id: I3cdde7bfab0bc0bea1993dd549b9b0e8d36db9be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch adds Intel NVMe device list and overrides the
supported log pages according to the quirk list.
In particular, the READ_CMD_LATENCY and WRITE_CMD_LATENCY pages are
supported on Intel DC P3x00 devices despite not being listed in the
Intel vendor-specific log page directory.
Change-Id: I3a2b6a5fa142c6e9c93567df65e85980bd3c7cc0
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Also add a space between Copyright and (c).
The copyright year can be determined using git metadata.
Also remove the duplicated "All rights reserved." - every instance of
this line already has a corresponding "All rights reserved" immediately
below it, except for examples/ioat/kperf/kmod/dma_perf.c, where I have
added it manually.
Performed using this command:
git ls-files | xargs sed -i -e 's/Copyright(c) \(.*\) Intel Corporation. All rights reserved./Copyright (c) Intel Corporation./'
Change-Id: I3779f404966800709024eb1eb66a50068af2716c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
1 Add new API nvme_ctrlr_is_feature_supported().
2 Add unit test for new API.
Change-Id: Ia6d8710755c3b13984fca9d56700efe043be1402
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
This will allow replacing these _nvme_fail_request_bad_vtophys() calls
with the correct error later. vtophys is not actually used within the
SGL request builder, so this is the wrong error.
Change-Id: Ibc2a3b029a8abad1d563b9df200325d7d64498da
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
No code change, just moved into a function for readability.
Change-Id: I883443c06d961c6dbeffed1a6fb153177e6e3fcd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This cleans up the I/O splitting code somewhat.
It also moves the SGL payload function pointers up into the hot cache
section of struct nvme_request without pushing the other important
members past the cacheline boundary (because payload is now a union).
Change-Id: I14a5c24f579d57bb84d845147d03aa53bb4bb209
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
A namespace indicates support for reservations by reporting a non-zero
value in the Reservation Capabilities field in the Identify Namespace
data structure, and controller indicates support for reservation in the
Identify Controller data structure, Here we used namespace field as the
support flag.
Change-Id: I0e1e29548aa3fc8b6d3bbeb4149ec4864316f092
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Support for the Force Unit Access and Limited Retry
bits on reads and writes.
Change-Id: I9860848358377d63a967a4ba6ee9c061faf284d4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
NVMe reservations provide capabilities that may be used by two or more
hosts to coordinate access to a shared namespace, here we add the 4
reservation commands: reservation register/acquire/release/report.
Change-Id: Ib03ae2120a57dd14aa64311a6ffeb39fda73018c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
For the purpose to support different types of input scattered payloads,
such as iovs or scattered list, we define common method in the NVMe
driver, users should implement their own functions to iterate each
segment memory.
Change-Id: Id2765747296a66997518281af0db04888ffc4b53
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change nvme_ctrlr_is_log_page_supported() to match
nvme_ctrlr_cmd_get_log_page().
Change-Id: I4c8a1f11044b083f8f8990ef40a4f789fa3c24e3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Request allocation may fail, so we need a way to indicate failure to the
caller.
Change-Id: I278c3f42e4d2fa1902bb0ab33ad3bf7c7007fd0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
1 Add supported log pages data structure.
2 Bulid up supported log pages when NVME start.
3 Provide unified API for getting log pages.
3 Unit test suit optimization base on above modification.
Change-Id: I03cdb93f5c94e6897510d7f19bc7d9f4e70f9222
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
nvme_ctrlr_process_io_completions() and
nvme_ctrlr_process_admin_completions() now return the number of
completions processed.
This also adds the possibility of returning an error from the
process_*_completions functions (currently unused, but this at least
gets the API ready in case error conditions are added later).
Change-Id: I1b32ee4f2f3c1c474d646fa2d6b8b7bbb769785f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Previously, if nvme_allocate_request() failed in
nvme_ctrlr_construct_and_submit_aer(), there was no error checking, so a
NULL pointer would be dereferenced.
Add a return value to nvme_ctrlr_construct_and_submit_aer() so we can
signal failure to the caller. This can only really be reasonably
handled during initialization; when resubmitting a completed AER later,
there is nowhere to report failure, so the AER will just remain
unsubmitted.
Change-Id: I413eb6c21be01cd9a61e67f62f2d0b7170eabaa3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
According to the specification, the dataset management for deallocate
attribute can support to 256 ranges, so we should use uint16_t
instead of uint8_t as the ranges parameter.
Change-Id: Ibacc00da8b4b9e2b2f3454d382aadf7ad353ff31
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The parent field is no longer used in the normal (non-split) I/O path,
so move it down to the default-uninitialized part of struct nvme_request
that is only touched for parent/child I/O.
This also puts it closer to other related fields (children,
child_tailq, parent_status) for improved readability.
Change-Id: I120df1df0c967d2f74daa6e97c0bc83626e3be7f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_qpair_submit_tracker() and nvme_qpair_manual_complete_request() are
only used from within nvme_qpair.c, so they can be static.
nvme_qpair_submit_tracker() is moved up to avoid needing a declaration
(no other code change).
nvme_ctrlr_hw_reset() is only used from within nvme_ctrlr.c, so it can
be static.
Change-Id: I9a7953d7baaec76e875dd535daf557ea24bef801
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These delays are left over from early development. They are completely
unnecessary and not based on anything in the NVMe spec.
Startup time should be slightly improved (on the order of 100 ms in
normal cases).
Change-Id: I9068b1a0f42feabcfe656d68be91e05a56cc53a3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
NVMe doesn't require the specific 64-bit MMIO ordering on 32-bit
platforms performed in spdk_mmio_read_8(), but it doesn't hurt.
We have to pick one of the two possible orderings, so pick the one
required by I/OAT.
Change-Id: I2b909d64d0c077b797d0f64a11d78d1ecc55eec7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use $(CLEAN_C) throughout the Makefiles to clean up a consistent set of
generated files.
This also adds coverage files to the list of cleaned files.
Change-Id: Iceb922935a45c9eecbf2f3443bd0ee4f5c966825
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Most devices today support far fewer than 1024, but this is a
more reasonable default upper limit than the spec-defined 64K.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia8a6d80c3a5aa181f27c8354758c6ca468013d92
lib/memory was already using this pattern; extend it to lib/util and
lib/nvme.
Change-Id: I84a633d7805522fc94d8fc11ad5486ce552702e5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The .o files are always kept anyway, so there is no need for an explicit
rule.
Change-Id: Id1687ba89daabfda5802e4328deb127403277928
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add common $(COMPILE_C), $(LINK_C), and $(LIB_C) variables that contain
the commands to build a .o from a .c, an app from objects and libraries,
and a library from objects, respectively.
Change-Id: Ie2eaa13156b8bf3db7a4ffa66161382d829aef07
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_ctrlr_process_io_completions() now takes a second parameter,
max_completions, to let the user limit the number of I/Os completed on
each poll.
If there are many I/Os waiting to be completed, the
nvme_ctrlr_process_io_completions() function could run for a long time
before returning control to the user, so the max_completions parameter
lets the user have more control of latency.
Change-Id: I3173059d94ec1cc5dbb636fc0ffd3dc09f3bfe4b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
After converting is_resetting to bool, it is smaller and can be packed
more efficiently with is_failed and reordered after the larger fields
used in the I/O path.
Change-Id: Ifa2301eb61ce8d38eb5412cca61d2a91b1474101
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It was previously uint32_t because it was accessed with special
uint32_t-only atomic read/write helper functions, but that was replaced
with normal variable accesses protected by a mutex.
Change-Id: I304a7ef8c723cb33fd08110b697f848823a163e7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Set SPDK_ROOT_DIR explicitly in each Makefile so that make from a
subdirectory will work (assuming all dependencies from the upper
directory have already been built). This allows partial rebuilds of the
source tree, as well as building the unit tests without requiring DPDK.
Change-Id: I3f65b805d490b40ff5ec53cceb61df542ce814f1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fix all of the uses of __thread so they are at the beginning (similar to
e.g. static).
Don't actually enable -Wold-style-declaration, since clang doesn't
understand that.
Change-Id: I0dcbb758143eab90fc978334c8f256c6602cc4cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename the nvme_free_request macro to nvme_dealloc_request to match
nvme_alloc_request and add a wrapper function to nvme.c so that the
macro contents are only expanded once.
The DPDK nvme_impl.h uses rte_mempool_put(), which generates a large
amount of code inline. Moving this macro expansion to a wrapper
function avoids inlining it in the multiple places nvme_free_request()
gets called, most of which are error handling cases that are not in the
hot I/O path.
Change-Id: I64ea9c39ba47e26672eee8d5058f1489e07eee5b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is the only place nvme_request_add_child() is used, so move it
nearby and make it static to allow the compiler to inline it.
Change-Id: If4a7e17fde0b0272e1d4432c1dcedbec27c25371
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Move dependency includes into a new spdk.deps.mk file,
then include it at the end of Makefiles that build
source files.
Also add a test to autobuild.sh to confirm that
binaries are regenerated if we make after touching a
header file.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If6a1905706a840f92cbdf3ace7fbdb27fe2de213
Pull the almost-identical request splitting code for driver-assisted
striping and maximum I/O size into its own function,
_nvme_ns_cmd_split_request().
Change-Id: I3c15ac2073f8f5aec721c427199c8fb1a5d6a1fc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
GCC generates a series of 64-bit MOV instructions for the memcpy() into
the submission queue. We can do better with 128-bit SSE2 instructions.
DPDK already has a memcpy implementation that is optimized for small
inline copies, so use it instead of memcpy.
Change-Id: I5f09259b4d5cb089ace4a8ea6d2078c03fee84f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
No change in behavior, just a simplification.
We already have a check for retry, so pull the cb_fn check out and put
it under the !retry branch.
This makes it clearer that requests that are going to be retried will
not get their callbacks called.
Change-Id: I70c7067e550c7fca78b0441b5474833f73863315
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Since it is empty function. It can be deleted.
Add nvme_assert to fix issue reported by scan-build.
Change-Id: Ia0e8f656e1dac0da7ec72f8404469ea1b0dcb40e
Signed-off-by: Liang Yan <liangx.yan@intel.com>
This is the only place that was using printf directly in the NVMe
library. Replace it with the official nvme_printf logging mechanism.
Change-Id: I689a7c0854b5e47eb357150f814e347cd44be79c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
In nvme_qpair_complete_tracker, make sure we got a valid request in the
tracker that is being completed.
This should never occur in practice, but safeguard against it in case of
programmer error. Fixes a scan-build warning about potential NULL
dereference.
Change-Id: Id82af604d2a5ed5de0aeccf3affa1900f6712ebe
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Document the assumptions made by qpair_construct using asserts.
These values can't actually be 0 in practice due to the way they are
derived, but scan-build can't see that. It is also useful to have these
asserts in case of future modifications.
Change-Id: I546c057f5cbe7ccc62acd90b595e423cd450d86a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_dump_command is totally unused aside from the unit test.
nvme_dump_completion was used in qpair, but it can be replaced with the
equivalent nvme_qpair_print_completion.
Also added the missing nvme_completion fields to nvme_qpair_print_completion
that had been printed by nvme_dump_command.
Change-Id: Ia5ee66f3553df06febe8f465d42e49a84c555dd2
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is calculated elsewhere now, so remove the comments around
nvme_qpair_construct calls.
Change-Id: I2dc4956a9e250b88e62038bc55cdd315940ad391
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
rc is reinitialized before it is ever read.
Change-Id: I9abbc256fb06022f3024b0aa3827be02a273f20a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
