When multiple NVMe controllers are being initialized during
spdk_nvme_probe(), we can overlap the hardware resets of all controllers
to improve startup time.
Rewrite the initialization sequence as a polling function,
nvme_ctrlr_process_init(), that maintains a per-controller state machine
to determine which initialization step is underway. Each step also has
a timeout to ensure the process will terminate if the hardware is hung.
Currently, only the hardware reset (toggling of CC.EN and waiting for
CSTS.RDY) is done in parallel; the rest of initialization is done
sequentially in nvme_ctrlr_start() as before. These steps could also be
parallelized in a similar framework if measurements indicate that they
take a significant amount of time.
Change-Id: I02ce5863f1b5c13ad65ccd8be571085528d98bd5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This check was dead code, since both places that called
nvme_ctrlr_wait_for_ready() could only ever have cc.en = 1.
Remove the original nvme_ctrlr_wait_for_ready() wrapper and rename
_nvme_ctrlr_wait_for_ready() without the underscore to replace it.
Change-Id: I6c9aa6a5b93606fb89d168c23f6735fcf3a84eaa
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
In nvme_ctrlr_hw_reset(), if we encounter a controller whose CC.EN bit
is already 0 (controller is disabled), the previous code would enable
the controller just so that it could be disabled to get a full reset
(transition from CC.EN = 1 to CC.EN = 0). However, it is a safe
assumption that if CC.EN is already 0, the controller has just been
reset, so we don't need to reset it again.
This saves a significant amount of time (2+ seconds per controller with
Intel SSD DC P3700) during initialization for devices that were disabled
on startup.
Change-Id: I552b1f0f185a84a8a0ce57a93b012d9d5fe096f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
cleanup.sh and unbind.sh have been combined into a single
setup.sh that takes one optional parameter (reset). If no
parameter is given, the script will automatically bind
all NVMe and IOAT devices to either uio_pci_generic
or vfio-pci, as appropriate based on IOMMU settings. If
the reset parameter is given, the devices will be bound back
to the appropriate kernel drivers.
Change-Id: I25db3234f1ecfb352a281e5093f4c1aa455152ae
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Explicitly include system headers for types that are used in public
headers.
These were being pulled in by example code, so SPDK itself would build,
but other apps that did not include stdbool.h would fail to compile when
including spdk/nvme.h.
Also include nvme.h first in nvme_internal.h so this case gets tested
during normal compilation.
Change-Id: I8ed0fc4e0dcf71551738c461b4b825cc2ee1d233
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
payload here is a pointer to the buffer, not a struct nvme_payload.
Use nvme_allocate_request_contig() and pass the length in bytes rather
than dwords.
Change-Id: Idbbb3614b1d69148fe041d26e0c148bd9ce53724
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The parent field is in the cache line of nvme_request that is only
supposed to be accessed for split (child) I/Os. All accesses to parent
are done from child-specific calls now, so it does not need to be
initialized in the common case of a non-split I/O.
nvme_request_add_child() will set parent when splitting occurs.
Change-Id: Ib86c16ba1ea2ce32f62079831101da2a099047af
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Allows simplification of _nvme_qpair_build_sgl_request(), which does not
need to know whether a request is a child or not.
This also removes a read of req->parent for non-split I/Os; the parent
field is in the section of nvme_request that is only intended to be
initialized for split I/Os, which should be detected by looking at
num_children.
Additionally, this fixes a potential problem if requests were nested
more than one level deep (e.g. req->parent was not the original user
request).
Change-Id: I3ea1dc134bbb1e3b8c6b5a479f5d760bd97ea848
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch fixes the following issues in ioat_impl.h
while CONFIG_PCIACCESS is set to n:
1 lack of the definition of ioat_pcicfg_unmap_bar.
2 Define a macro to remove the duplicated vendor code
for ioat_driver_id enumration.
Change-Id: I4011098ac296d1ec320bffb5ffa6e098b70a5545
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Building the tip of the spdk master against the dpdk-2.2.0 fails with
inappropriate RTE_CACHE_LINE_SIZE error. The simple reversal of the RTE
include file order below fixed it for me.
Change-Id: I8782b7ee21d7f185e6e678f874fbdab9403117a5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch is used to remove the duplicated code.
As we found the structure in "ifdef and else" are same.
Change-Id: I1717ce3dcc14134ac59c165d801e5e811b987be5
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
This patch is used to support spdk_pci_device_cfg read/write
with byte and word size.
Change-Id: I49084e231bd6b5f5b22180a3eb36ddad4430b3a4
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Move toward collecting PCI IDs, class codes, etc. in pci_ids.h instead
of individual device-specific headers.
Change-Id: Icff162d48ac663db71d0576ceee16a9bd7a751cd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
PCI_VENDOR_ID_INTEL -> SPDK_PCI_VID_INTEL
Also change the inclusion guard macro to be consistent with the other
SPDK headers.
Change-Id: I29346267172cb8c07cc4289eed4eca2d55e942d6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This doesn't need to be part of the public API. It is only used by the
NVMe quirk lookup tables.
Change-Id: I7662e333c70b7c5f814bd7c8a528b6bff1f0732e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename all functions with a spdk_ prefix, and provide enough of an API
to avoid apps needing to #include <pciaccess.h>.
The opaque type used in the public API for a PCI device is now
struct spdk_pci_device *.
Change-Id: I1e7a09bbc5328c624bec8cf5c8a69ab0ea8e8254
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is a step toward abstracting PCI access so that libpciaccess can be
swapped out more easily.
Change-Id: I5491459460cbfbd0be471f70f9d07a7eb3175234
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Instead of writing the completion doorbell once per completion,
just write it once at the end of the completion while loop.
This reduces the number of mmio writes by coalescing several
writes into one when we get multiple completions at a time.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3cc3864dcfe43186bec51be1a732e84ef3be05ae
Similar to the NVMe API change, this allows better abstraction of the
PCI subsystem.
Change-Id: I2b84d9c3c498a08d4451b4ff27d0865f0456c210
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch did the work to use pci related functions
provided by DPDK.
Change-Id: I263b79f1b42868ef0c1efcf1bc392a4b3a328e93
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
With CONFIG_PCIACCESS=y in CONFIG file, we can
use libpciaccess library; With CONFIG_PCIACCESS=n
in CONFIG file, we use pciaccess functions provided
in DPDK.
Change-Id: I786c5589b8e7909ba2e59d222938dd5ba45bf92d
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
The new probing API will find all NVMe devices on the system and ask the
caller whether to attach to each one. The caller will then receive a
callback once each controller has finished initializing and has been
attached to the driver.
This will enable cleanup of the PCI abstraction layer (enabling us to
use DPDK PCI functionality) as well as allowing future work on parallel
NVMe controller startup and PCIe hotplug support.
Change-Id: I3cdde7bfab0bc0bea1993dd549b9b0e8d36db9be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For those Crystal Beach DMA channels which support block fill capability,
we add a fill API here that can zero out pages or fill them with a
fixed pattern.
Change-Id: I8a57337702b951c703d494004b111f6d206279fb
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
This patch adds Intel NVMe device list and overrides the
supported log pages according to the quirk list.
In particular, the READ_CMD_LATENCY and WRITE_CMD_LATENCY pages are
supported on Intel DC P3x00 devices despite not being listed in the
Intel vendor-specific log page directory.
Change-Id: I3a2b6a5fa142c6e9c93567df65e85980bd3c7cc0
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Also add a space between Copyright and (c).
The copyright year can be determined using git metadata.
Also remove the duplicated "All rights reserved." - every instance of
this line already has a corresponding "All rights reserved" immediately
below it, except for examples/ioat/kperf/kmod/dma_perf.c, where I have
added it manually.
Performed using this command:
git ls-files | xargs sed -i -e 's/Copyright(c) \(.*\) Intel Corporation. All rights reserved./Copyright (c) Intel Corporation./'
Change-Id: I3779f404966800709024eb1eb66a50068af2716c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
1 Add new API nvme_ctrlr_is_feature_supported().
2 Add unit test for new API.
Change-Id: Ia6d8710755c3b13984fca9d56700efe043be1402
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
This will allow replacing these _nvme_fail_request_bad_vtophys() calls
with the correct error later. vtophys is not actually used within the
SGL request builder, so this is the wrong error.
Change-Id: Ibc2a3b029a8abad1d563b9df200325d7d64498da
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
No code change, just moved into a function for readability.
Change-Id: I883443c06d961c6dbeffed1a6fb153177e6e3fcd
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This cleans up the I/O splitting code somewhat.
It also moves the SGL payload function pointers up into the hot cache
section of struct nvme_request without pushing the other important
members past the cacheline boundary (because payload is now a union).
Change-Id: I14a5c24f579d57bb84d845147d03aa53bb4bb209
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
A namespace indicates support for reservations by reporting a non-zero
value in the Reservation Capabilities field in the Identify Namespace
data structure, and controller indicates support for reservation in the
Identify Controller data structure, Here we used namespace field as the
support flag.
Change-Id: I0e1e29548aa3fc8b6d3bbeb4149ec4864316f092
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Support for the Force Unit Access and Limited Retry
bits on reads and writes.
Change-Id: I9860848358377d63a967a4ba6ee9c061faf284d4
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This patch is used to wrap the inner implementation
of libpciaccess and prepare for the same interface
to applications in the future patch
Change-Id: I4d40fae0bd86b451ed38dbfd9bcc015f9bfc8436
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
NVMe reservations provide capabilities that may be used by two or more
hosts to coordinate access to a shared namespace, here we add the 4
reservation commands: reservation register/acquire/release/report.
Change-Id: Ib03ae2120a57dd14aa64311a6ffeb39fda73018c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
For the purpose to support different types of input scattered payloads,
such as iovs or scattered list, we define common method in the NVMe
driver, users should implement their own functions to iterate each
segment memory.
Change-Id: Id2765747296a66997518281af0db04888ffc4b53
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Change nvme_ctrlr_is_log_page_supported() to match
nvme_ctrlr_cmd_get_log_page().
Change-Id: I4c8a1f11044b083f8f8990ef40a4f789fa3c24e3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Request allocation may fail, so we need a way to indicate failure to the
caller.
Change-Id: I278c3f42e4d2fa1902bb0ab33ad3bf7c7007fd0d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
