Commit Graph

98 Commits

Author SHA1 Message Date
Changpeng Liu
8181c59293 nvme: override the default queue depth if users specified the value
SPDK already uses DEFAULT_IO_QUEUE_SIZE and MQES to decide the correct
queue depth of NVMe queue pair, hardcoded it to NVME_IO_ENTRIES(512)
does not make sense if users want to set queue depth bigger than 512.

Change-Id: Iaa73fc79e055292ae9bd19af0c8c12f257ae5c46
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/379052
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-09-20 19:58:49 -04:00
Changpeng Liu
38c19cd275 nvme: change PSDT to SGL_MPTR if hardware SGL is supported
Use SGL with contiguous physical buffer metadata as the default PSDT
if NVMe can support SGL.

Change-Id: I99192a8b5408f1b034d5112d77d13639b7558f01
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/378699
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ziye Yang <optimistyzy@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-09-20 19:55:53 -04:00
Changpeng Liu
26089992fe nvme: print the error message when error happened
Users don't need to provide nvme debug flag for the error cases.

Change-Id: I00c29e2b8ab470b0233d94acec52b4bec129728c
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/376708
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-09-06 18:11:24 -04:00
Daniel Verkamp
d92f0f75ca log: rename SPDK_TRACELOG to SPDK_DEBUGLOG
This matches the name to the behavior and prepares for addition of a new
log macro for "info" log level.

Change-Id: I94ccd49face4309d3368e399528776ab140748c4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/375833
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-08-29 13:25:58 -04:00
Daniel Verkamp
2eec131e4d nvme: decouple NVMe memory page size from PAGE_SIZE
PAGE_SIZE is the host memory page size, which is irrelevant for the NVMe
driver; what we actually care about is the NVMe controller's memory page
size, CC.MPS.

This patch cleans up the uses of PAGE_SIZE in the NVMe driver; the
behavior is still the same in all cases today, since normal NVMe
controllers report a minimum page size of 4096.

Change-Id: I56fce2770862329a9ce25370722f44269234ed46
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/374371
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2017-08-17 12:52:12 -04:00
Daniel Verkamp
269910c05c nvme: refactor PRP building code
This also changes the SGL -> PRP case to translate each 4K page from
virtual to physical, in case the buffer is not physically contiguous.

Change-Id: If027f9d656c52c56504f0c64cd4464e16440df63
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/371616
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2017-08-08 14:09:33 -04:00
Jim Harris
002660c4f0 nvme: add transport get_max_sges abstraction
For pcie, this just equals the number of SGLs we can fit
into the per-tracker memory.

For rdma, this is just set to 1 for now since nvme_rdma.c
does not support multiple SGEs yet.  Once that support is
added, this will change to use MSDBD (Maximum SGL Data Block
Descriptors) instead from the controller identify data.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I34a4c546b5ff46918a296a73ed8cbcc6c9879d5a

Reviewed-on: https://review.gerrithub.io/372358
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2017-08-02 17:15:21 -04:00
Daniel Verkamp
ce4fcbce71 nvme: add I/O qpair creation options
Add a new struct spdk_nvme_io_qpair_opts to allow the user to override
controller options on a per-I/O qpair basis.

Existing callers with qprio == 0 can be updated to:

  ... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0);

Callers that need to specify a non-default qprio should be updated to:

  struct spdk_nvme_io_qpair_opts opts;
  spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts));
  opts.qprio = SPDK_NVME_QPRIO_...;
  ... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts));

Change-Id: I8ac3ea369535cfde759abbe75e1d974b6450a800
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/369676
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2017-07-18 18:33:20 -04:00
GangCao
619ab1a700 nvme: no need to send abort notification for died application
Change-Id: I7005159b2c4cc71c249cfdc4adbf8500f006fb1c
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/365646
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-06-20 15:43:21 -04:00
Ben Walker
89ad3f3d6b nvme: Abort outstanding commands when qpair is deleted
These commands should be treated as aborted by spec,
so correctly deliver abort notifications when a
qpair is deleted.

Change-Id: I8af47a3f42f5695ef8e1a70813662e69102720b2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/364681
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-06-09 13:49:13 -04:00
John Meneghini
8a44220b1a env: Rename spdk_malloc/zmalloc/realloc/free to spdk_dma_(func)
- rename spdk_malloc_socket to spdk_dma_malloc_socket
  - rename spdk_malloc to spdk_dma_malloc
  - rename spdk_zmalloc to spdk_dma_zmalloc
  - rename spdk_realloc to spdk_dma_realloc
  - rename spdk_free to spdk_dma_free

Change-Id: I52a11b7a4243281f9c56f503e826fd7c4a1fd883
Signed-off-by: John Meneghini <johnm@netapp.com>
Reviewed-on: https://review.gerrithub.io/362604
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2017-05-31 15:30:27 -04:00
GangCao
afbb5ceec2 nvme: remove NVME_IO_TRACKERS to use hardware and user specified size
Change-Id: Ie6946343314a62e4416d8cd7dadd98c20a1c1e3a
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/361700
Tested-by: <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2017-05-23 16:01:52 -04:00
GangCao
b347d551e8 nvme: detach the pci device with calling process's own devhandle
Change-Id: I2693b4bd29e0500379d5e399723aec7e44bffca3
Signed-off-by: GangCao <gang.cao@intel.com>
2017-05-16 10:35:29 -07:00
Ben Walker
b961d9cc12 include: Move the remainder of the code base to stdinc.h
Change-Id: I6a142feeaad3117bd3c75e7c5cb7231a1cfa78ae
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-05-08 13:20:36 -07:00
Jim Harris
fd5d6901f2 nvme: use spdk_likely/unlikely macros in I/O path
Also avoid an spdk_get_ticks() call in the default
case where a timeout_cb_fn is not defined.

On my Intel(R) Xeon(R) E5-2699 v3 system with an
Intel(R) P3700 SSD, these modifications reduce software
overhead per I/O by 3-5% (as measured by the SPDK
overhead tool).

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5bb5a87747b15d9e27655fabcd2bc1a40b0b990e
2017-04-17 11:56:36 -07:00
Daniel Verkamp
5742e9b9e7 nvme: allocate requests on a per-queue basis
Change-Id: I7bec816e518a0a6f2e9fb719128c83d4b908d46c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-04-12 21:05:24 -07:00
Daniel Verkamp
cd13f280f4 nvme: track qpair within nvme_request
Change-Id: Ia40a1e79db6327c4693731e9bb7a57810795429d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-04-12 21:05:23 -07:00
Ben Walker
ee460db71f nvme: Don't pass admin qpair through timeout callback
Instead, pass NULL when an ADMIN command times out.
We don't expose the admin queue to the user.

Change-Id: If0768d329a689f6f7c3734c9d419e680d7378ed1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-04-12 13:11:01 -07:00
Ben Walker
b9feeb03c1 nvme: Call the timeout callback only a single time
For each command that times out, call the timeout
callback one time if the user registered one.

Change-Id: Iaad39a886468e89bef63fe292c5cad1dce97a57c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-04-12 13:11:01 -07:00
Ben Walker
2e82b58684 nvme: Release lock before calling remove_cb
This prevents a deadlock if the user immediately
calls spdk_nvme_detach.

Change-Id: I79f28abe163cbbf184bea907692c44aa4e1c8893
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2017-03-29 21:07:21 -07:00
Daniel Verkamp
2563429809 nvme: convert uevent failure message to TRACELOG
The message about the uevent socket is not a fatal error; it just means
that hotplug monitoring will not work.

Change-Id: I29f6a253e96a86420c0fde9e19135f9f1d229bb9
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-03-21 15:39:21 -07:00
Cunyin Chang
b7349216b9 nvme: optimization for nvme probe and detachment.
remove the unnecessary rte_eal_pci_probe_one() in function
spdk_pci_device_detach(), this could cause error message when we
terminate the application, it will also not make sense try to probe one
device after we detach it, we could call spdk_pci_nvme_device_attach()
instead of spdk_pci_nvme_enumerate() when we have one given device address,
dpdk will try to scan the device and add it back to pci device list then.

Change-Id: I35f5bb412249bb20da57394f0531c10a49691906
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2017-03-15 09:00:25 -07:00
Daniel Verkamp
b6d6cd8e81 nvme: pass command ID to timeout callback
This can be used for issuing an abort for the timed-out command.

Change-Id: I3c5727fdddc156cd7c8f99afbc3e6da8e73bba56
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-02-23 17:49:01 -07:00
Daniel Verkamp
5f2f2052c0 nvme: reorder spdk_nvme_timeout_cb arguments
Move cb_arg to the first argument to match the other NVMe callback
function signatures.

Change-Id: I4e699c8071dcb7ba4ce3cdb82ee985600208204c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-02-22 17:55:43 -07:00
Daniel Verkamp
84d904841f util: move common helper functions to util.h
These were repeated a few different places, so pull them into a common
header file.

Change-Id: Id807fa2cfec0de2e0363aeb081510fb801781985
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-02-10 09:38:52 -07:00
Daniel Verkamp
ff0047816d nvme: track submit time instead of timeout time
This removes one addition from the submission path (negligible, but a
nice side effect), but also opens up the possibility of reporting the
total time an I/O took - since we are always tracking the submission
time anyway, there is no extra cost to report it in the completion
callback.

Change-Id: I7129e7c09d20da8082042a7622d045846461dd9c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-02-08 09:23:58 -07:00
Daniel Verkamp
018e6afff2 nvme: remove ctrlr_get_pci_id transport callback
The PCIe transport initializes the quirks directly, so the generic hook
to get PCI ID is no longer necessary.  This path was dead code.

Change-Id: I25bdaa598db53e4312a264d9d8356d1b416696e5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-01-31 14:50:40 -07:00
Daniel Verkamp
a384110d7a nvme: move nvme_qpair_fail() call to generic code
The logic to fail queue pairs when the controller is failed should be
handled in the generic code, not in the individual transports.

This also allows nvme_qpair_fail() to be private to nvme_qpair.c.

Change-Id: I6194576dceb35073b9af8847e59314900028637c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-01-31 14:50:40 -07:00
Cunyin Chang
5f3e922a6a nvme: Fail the qpair if it is marked as failure when process the completion queue.
This make sure the qpair failure could be started from upper level application.

Change-Id: I7e04fe36929cc634ddf0078db96fbc40afb38f8c
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2017-01-25 16:31:52 -07:00
Cunyin Chang
2d5087b305 nvme: Add assert for g_thread_mmio_ctrlr in sigbus error handler function.
The g_thread_mmio_ctrlr should be not NULL pointer when it enter the
handler function.

Change-Id: I45dba601c672b16e2c6feafd9059bafde0d8f1b4
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2017-01-23 16:10:09 -07:00
Daniel Verkamp
d63a30e39d nvme/pcie: return 1 when PCI address doesn't match
If the user asked for a specific PCI address in spdk_nvme_probe(), we
need to return 1, not 0, for the other PCI addresses that don't match
when enumerating.  0 means to attach the PCI driver, whereas 1 means to
continue enumerating.

With the previous behavior of returning 0, all NVMe devices would be
attached to the DPDK PCI driver, even if the user did not request for
them to be probed, and further calls to spdk_nvme_probe() would not find
any devices.

Change-Id: Ifbbcd7d1abe8ab535b6957855172e66a3e69fbe4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-01-20 17:07:30 -07:00
Ziye Yang
0a573526b6 nvme/pcie: Add the support to probe nvme by pci_addr
Currently we use the pci functions provided by DPDK,
it identifies the device by class id related
info but not by pci bdf info, so we can add the filering
by pci_addr in pcie_nvme_enum_cb function.

Change-Id: I5942e98853f00fc10fa6aae5c113517653d1b357
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
2017-01-18 15:30:45 -07:00
Jim Harris
86e8a920bf nvme: split non-compliant SGLs into multiple requests
Since nvme_ns_cmd.c now walks the SGL, some of the test code
needs to also be updated to initialize and return correct values
such as ctrlr->flags and sge_length.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I521213695def35d0897aabf57a0638a6c347632e
2017-01-17 07:51:09 -07:00
Daniel Verkamp
f80c0f4fdd nvme: remove transport ctrlr_attach callback
Now that the hotplug code is isolated in nvme_pcie.c, it can call the
PCIe transport attach function directly.

Change-Id: I2df3b9168473b537cc9b13367e06d3d3b6fa22be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-01-09 11:36:27 -07:00
Daniel Verkamp
df8129fb39 nvme: move num_entries to transport-specific qpairs
The spdk_nvme_qpair::num_entries value is never used in the common code,
so move it to the individual transport qpairs to make it clear that it
is a transport-specific implementation detail.

Change-Id: I5c8f0de4fcd808912ba6d248cf5cee816079fd32
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2017-01-05 15:49:09 -07:00
Cunyin Chang
6152d5bf8f nvme: move hot-plug logic to nvme pcie layer.
Change-Id: I2042b34e4284a9c59aa3092ccd061c075748880c
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
2017-01-03 16:56:13 -07:00
Ben Walker
a96a6ecf58 nvme: Rename nvme_qpair_construct to nvme_qpair_init
This function initializes the members of an existing
qpair struct. It doesn't construct one from scratch.

Change-Id: I0b9afac1ad25cfb217efd146702f693c74f5f697
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-19 12:51:50 -07:00
Daniel Verkamp
0eb3125531 nvme: remove transport qpair_construct callback
Make the qpair construct functions private to the transports - it
doesn't need to be called from generic code.

Change-Id: I5f730a4bcf60ce231fe27bc8f4c3c39cb647dd2d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-12-15 11:09:49 -07:00
Daniel Verkamp
988906135c nvme: determine io_queue_size at startup
Add a transport callback to return the maximum queue size, and enforce
it in the generic nvme_ctrlr layer.

This allows the user to tell what io_queue_size was actually selected by
the transport via the ctrlr_opts returned during attach_cb.

Change-Id: I8a51332cc01c6655e2a3a171bb92877fe48ea267
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-12-15 11:09:48 -07:00
Daniel Verkamp
f505f57b36 nvme: add ctrlr function to set initial CAP value
Change-Id: Id7f68f279f87a08844015efcd35a790944b3bfbe
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-12-15 11:09:25 -07:00
Daniel Verkamp
92ee5fc4be nvme: rename queue_size option to io_queue_size
This better describes what the field controls (it does not affect the
admin queue size).

Change-Id: I851ae46fb4ed0fce819af07ae235824e0fc817e6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-12-13 15:23:13 -07:00
Ben Walker
ec0b5d2ba1 nvme: Clean up redundant arguments in transport_ctrlr_construct
The other simplifications to probe_info and trid made the
trtype argument redundant.

Change-Id: Ie7bea4e2204e690dc4909eeacd065e0722b53272
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-12 10:49:17 -07:00
Ben Walker
32e838af3c nvme: Remove probe_info, just use transport_id
The probe_info was reduced to just containing a
transport_id, so remove probe_info entirely.

Change-Id: Ica9a22d126cd14e282decd3eea1a0afe0460f099
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-12 10:49:17 -07:00
Ben Walker
6c2e170dc7 nvme: Remove pci_id from probe_info
This can be obtained by parsing traddr into a pci_addr,
then getting a handle to the pci_dev and asking for all
of the pci information.

Change-Id: I1948cbd3ec65611293192ef5558ace19dd444d4c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-12 10:49:13 -07:00
Ben Walker
3da43e64e4 nvme: Remove pci_addr from probe_info
This can be obtained by parsing the traddr.

Change-Id: Idaf35066cbf900c87e771a44934de99fb5420001
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-12 10:46:50 -07:00
Isaac Otsiabah
7c60aec01e nvme: add I/O timeout callback
Change-Id: Ie6220590467b88fe1e63b4b0f8a01221fc0c1206
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-12-10 16:17:18 -07:00
Isaac Otsiabah
a4cc346b17 nvme: convert nvme_tracker list to a TAILQ
Change-Id: I822529b28b683f707604c07721040f22689961c4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
2016-12-10 15:45:53 -07:00
Ben Walker
8a9c1d4011 nvme: Make striping a quirk
Use the standard quirk mechanism to specify which devices
need software assisted striping.

Change-Id: Id8156876a90b4caf9d687637e14c7ad4a66ceda6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-09 14:38:42 -07:00
Ben Walker
34100924e4 nvme: Rename nvme_probe_one to nvme_ctrlr_probe
Simplify the arguments as well.

Change-Id: I653e1aad4e574cf95d793b41b749cb19e0024c2b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-08 15:22:48 -07:00
Ben Walker
06557b0a4d nvme: Add remove callback to transport scan
Scanning the transport may result in both new
devices and removed devices, so pass the callback
for both operations.

Change-Id: I6f73dbe6fd7cf61575c354b43f8ae3e2a01e2965
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
2016-12-08 15:22:47 -07:00