Commit Graph

62 Commits

Author SHA1 Message Date
Jim Harris
547d523eb8 Clean up debug prints.
1) Consistently use device_printf.
2) Make dump_completion and dump_command into something more
    human-readable.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 22:17:10 +00:00
Jim Harris
dd433dd0fb Move common code from the different nvme_allocate_request functions into a
separate function.

Sponsored by:	Intel
Suggested by:	carl
Reviewed by:	carl
2013-03-26 22:13:07 +00:00
Jim Harris
237d2019e5 Change a number of malloc(9) calls to use M_WAITOK instead of
M_NOWAIT.

Sponsored by:	Intel
Suggested by:	carl
Reviewed by:	carl
2013-03-26 22:11:34 +00:00
Jim Harris
955910a916 Replace usages of mtx_pool_find used for admin commands with a polling
mechanism.

Now that all requests are timed, we are guaranteed to get a completion
notification, even if it is an abort status due to a timed out admin
command.

This has the effect of simplifying the controller and namespace setup
code, so that it reads straight through rather than broken up into
a bunch of different callback functions.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 22:09:51 +00:00
Jim Harris
43a3725688 Abort and do not retry any outstanding admin commands left over after
a controller reset.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 22:06:05 +00:00
Jim Harris
232e2edb6c Add the ability to internally mark a controller as failed, if it is unable to
start or reset.  Also add a notifier for NVMe consumers for controller fail
conditions and plumb this notifier for nvd(4) to destroy the associated
GEOM disks when a failure occurs.

This requires a bit of work to cover the races when a consumer is sending
I/O requests to a controller that is transitioning to the failed state.  To
help cover this condition, add a task to defer completion of I/Os submitted
to a failed controller, so that the consumer will still always receive its
completions in a different context than the submission.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:58:38 +00:00
Jim Harris
3d7eb41c1b Just disable the controller instead of deleting IO queues during detach.
This is just as effective, and removes the need for a bunch of admin commands
to a controller that's going to be disabled shortly anyways.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:48:41 +00:00
Jim Harris
74019d4b67 Set Pre-boot Software Load Count to 0 at the end of the controller
start process.

The spec indicates the OS driver should use Set Features (Software
Progress Marker) to set the pre-boot software load count to 0
after the OS driver has successfully been initialized.  This allows
pre-boot software to determine if there have been any issues with the
OS loading.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:42:53 +00:00
Jim Harris
be34f21609 Remove the is_started flag from struct nvme_controller.
This flag was originally added to communicate to the sysctl code
which oids should be built, but there are easier ways to do this.  This
needs to be cleaned up prior to adding new controller states - for example,
controller failure.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:19:26 +00:00
Jim Harris
02e3348484 Ensure the controller's MDTS is accounted for in max_xfer_size.
The controller's IDENTIFY data contains MDTS (Max Data Transfer Size) to
allow the controller to specify the maximum I/O data transfer size.  nvme(4)
already provides a default maximum, but make sure it does not exceed what
MDTS reports.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:16:53 +00:00
Jim Harris
cb5b7c1304 Cap the number of retry attempts to a configurable number. This ensures
that if a specific I/O repeatedly times out, we don't retry it indefinitely.

The default number of retries will be 4, but is adjusted using hw.nvme.retry_count.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:14:51 +00:00
Jim Harris
0d7e13ecb2 Pass associated log page data to async event consumers, if requested.
Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:08:32 +00:00
Jim Harris
2868353a57 When an asynchronous event request is completed, automatically fetch the
specified log page.

This satisfies the spec condition that future async events of the same type
will not be sent until the associated log page is fetched.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:05:15 +00:00
Jim Harris
0692579bf3 Add structure definitions and controller command function for firmware
log pages.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:03:03 +00:00
Jim Harris
0892778256 Add structure definitions and a controller command function for
error log pages.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:01:53 +00:00
Jim Harris
cf81529ce3 Create struct nvme_status.
NVMe error log entries include status, so breaking this out into
its own data structure allows it to be included in both the
nvme_completion data structure as well as error log entry data
structures.

While here, expose nvme_completion_is_error(), and change all of
the places that were explicitly looking at sc/sct bits to use this
macro instead.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 21:00:18 +00:00
Jim Harris
f37c22a3bd Make nvme_ctrlr_reset a nop if a reset is already in progress.
This protects against cases where a controller crashes with multiple
I/O outstanding, each timing out and requesting controller resets
simultaneously.

While here, remove a debugging printf from a previous commit, and add
more logging around I/O that need to be resubmitted after a controller
reset.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 20:56:58 +00:00
Jim Harris
48ce317898 By default, always escalate to controller reset when an I/O times out.
While aborts are typically cleaner than a full controller reset, many times
an I/O timeout indicates other controller-level issues where aborts may not
work.  NVMe drivers for other operating systems are also defaulting to
controller reset rather than aborts for timed out I/O.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 20:32:57 +00:00
Jim Harris
941433323c Add a tunable for the I/O timeout interval. Default is still 30 seconds,
but can be adjusted between a min/max of 5 and 120 seconds.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 20:02:35 +00:00
Jim Harris
12d191ec12 Add handling for controller fatal status (csts.cfs).
On any I/O timeout, check for csts.cfs==1.  If set, the controller
is reporting fatal status and we reset the controller immediately,
rather than trying to abort the timed out command.

This changeset also includes deferring the controller start portion
of the reset to a separate task.  This ensures we are always performing
a controller start operation from a consistent context.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 19:58:17 +00:00
Jim Harris
dbba74428b Add API for nvme consumers to access controller and namespace identify data.
Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 19:52:57 +00:00
Jim Harris
b846efd7ec Add controller reset capability to nvme(4) and ability to explicitly
invoke it from nvmecontrol(8).

Controller reset will be performed in cases where I/O are repeatedly
timing out, the controller reports an unrecoverable condition, or
when explicitly requested via IOCTL or an nvme consumer.  Since the
controller may be in such a state where it cannot even process queue
deletion requests, we will perform a controller reset without trying
to clean up anything on the controller first.

Sponsored by:	Intel
Reviewed by:	carl
2013-03-26 19:50:46 +00:00
Jim Harris
65c2474e6d Keep a doubly-linked list of outstanding trackers.
This enables in-order re-submission of I/O after a controller reset.

Sponsored by:	Intel
2013-03-26 18:45:16 +00:00
Jim Harris
5f1e251de6 Create a generic nvme_ctrlr_cmd_get_log_page function, and change the
health information log page function to use it.

Sponsored by:	Intel
2013-03-26 18:43:53 +00:00
Jim Harris
99d99f7408 Expose the get/set features API to nvme consumers.
Sponsored by:	Intel
2013-03-26 18:42:05 +00:00
Jim Harris
038a5ee403 Add an interface for nvme shim drivers (i.e. nvd) to register for
notifications when new nvme controllers are added to the system.

Sponsored by:	Intel
2013-03-26 18:39:54 +00:00
Jim Harris
0a0b08cc30 Enable asynchronous event requests on non-Chatham devices.
Also add logic to clean up all outstanding asynchronous event requests
when resetting or shutting down the controller, since these requests
will not be explicitly completed by the controller itself.

Sponsored by:	Intel
2013-03-26 18:37:36 +00:00
Jim Harris
990e741c18 Move controller destruction code from nvme_detach() to new nvme_ctrlr_destruct()
function.

Sponsored by:	Intel
2013-03-26 18:34:19 +00:00
Jim Harris
274b3a88fa Specify command timeout interval on a per-command type basis.
This is primarily driven by the need to disable timeouts for asynchronous
event requests, which by nature should not be timed out.

Sponsored by:	Intel
2013-03-26 18:31:46 +00:00
Jim Harris
879de69910 Explicitly abort a timed out command, if the ABORT command sent to the
controller indicates the command was not found.

Sponsored by:	Intel
2013-03-26 18:29:04 +00:00
Jim Harris
6cb0607039 Break out the code for completing an nvme_tracker object into a separate
function.

This allows for completions outside the normal completion path, for example
when an ABORT command fails due to the controller reporting the targeted
command does not exist.  This is mainly for protection against a faulty
controller, but we need to clean up our internal request nonetheless.

Sponsored by:	Intel
2013-03-26 18:27:22 +00:00
Jim Harris
448195e764 Add support for ABORT commands, including issuing these commands when
an I/O times out.

Also ensure that we retry commands that are aborted due to a timeout.

Sponsored by:	Intel
2013-03-26 18:23:35 +00:00
Jim Harris
d6f54866ea Add an internal _nvme_qpair_submit_request function, which performs
the submit action assuming the qpair lock has already been acquired.

Also change nvme_qpair_submit_request to just lock/unlock the mutex
around a call to this new function.

This fixes a recursive mutex acquisition in the retry path.

Sponsored by:	Intel
2013-03-26 18:20:11 +00:00
Jim Harris
aaf6b84a4e Make the DSM range count 0-based. Previously we were deallocating one more
LBA than we should have been.

Sponsored by:	Intel
2013-03-26 18:16:30 +00:00
Jim Harris
51a9feb9b0 Do not look at the namespace's thin provisioning field to determine if DSM
command is supported.  The two are not related.

Sponsored by:	Intel
2013-03-26 18:01:24 +00:00
David E. O'Brien
4b52061e17 Fix GCC build:
/usr/src/sys/modules/nvme/../../dev/nvme/nvme.c:211: warning: format '%qx' expects type 'long unsigned int', but argument 9 has type 'long long unsigned int' [-Wformat]
2013-03-07 22:54:28 +00:00
Jim Harris
43e35466de Revert r244549.
This change was originally intended to account for test kthreads under
the nvmecontrol process, but jhb indicated it may not be safe to
associate kthreads with userland processes and this could have
unintended consequences.

I did not observe any problems with this change, but my testing didn't
exhaust the kinds of corner cases that could cause problems.  It is not
that important to account for these test threads under nvmecontrol, so I
am just reverting this change for now.

On a related note, the part of this patch for <= 7.x fails compilation
so reverting this fixes that too.

Suggested by: jhb
2013-01-07 21:35:25 +00:00
Jim Harris
96790da4f4 Put kthreads under curproc so they are attached to nvmecontrol rather
than pid 0.

Sponsored by:	Intel
2012-12-21 19:13:48 +00:00
Jim Harris
91fe20e34d Map BAR 4/5, because NVMe spec says devices may place the MSI-X table
behind BAR 4/5, rather than in BAR 0/1 with the control/doorbell registers.

Sponsored by:	Intel
2012-12-18 23:27:18 +00:00
Jim Harris
e1e84e74c1 Simplify module definition by adding nvme_modevent to DRIVER_MODULE()
definition.

Submitted by:   Carl Delsey <carl.r.delsey@intel.com>
2012-12-18 22:10:40 +00:00
Jim Harris
4d6abcb19f Do not use taskqueue to defer completion work when using INTx. INTx now
matches MSI-X behavior.

Sponsored by:	Intel
2012-12-18 21:50:48 +00:00
Jim Harris
38ce9496fe Add PCI device ID for 8-channel IDT NVMe controller, and clarify that the
previously defined IDT PCI device ID was for a 32-channel controller.

Submitted by:	Joe Golio <joseph.golio@isilon.com>
2012-12-06 15:36:24 +00:00
Jim Harris
633c572996 Use callout_reset_curcpu to allow the callout to be handled by the
current CPU and not always CPU 0.

This has the added benefit of reducing a huge amount of spinlock
contention on the callout_cpu spinlock for CPU 0.

Sponsored by:	Intel
2012-10-31 23:44:19 +00:00
Gleb Smirnoff
9427a0feed Fix build after r241659. 2012-10-18 14:25:33 +00:00
Jim Harris
0f71ecf741 Add ability to queue nvme_request objects if no nvme_trackers are available.
This eliminates the need to manage queue depth at the nvd(4) level for
Chatham prototype board workarounds, and also adds the ability to
accept a number of requests on a single qpair that is much larger
than the number of trackers allocated.

Sponsored by:	Intel
2012-10-18 00:45:53 +00:00
Jim Harris
21b6da584b Preallocate a limited number of nvme_tracker objects per qpair, rather
than dynamically creating them at runtime.

Sponsored by:	Intel
2012-10-18 00:44:39 +00:00
Jim Harris
5ae9ed6811 Create nvme_qpair_submit_request() which eliminates all of the code
duplication between the admin and io controller-level submit
functions.

Sponsored by:	Intel
2012-10-18 00:43:25 +00:00
Jim Harris
c2e83b404f Simplify how the qpair lock is acquired and released.
Sponsored by:	Intel
2012-10-18 00:41:31 +00:00
Jim Harris
5fa5cc5f12 Cleanup uio-related code to use struct nvme_request and
nvme_ctrlr_submit_io_request().

While here, also fix case where a uio may have more than 1 iovec.
NVMe's definition of SGEs (called PRPs) only allows for the first SGE to
start on a non-page boundary.  The simplest way to handle this is to
construct a temporary uio for each iovec, and submit an NVMe request
for each.

Sponsored by:	Intel
2012-10-18 00:40:40 +00:00
Jim Harris
d281e8fbbd Add nvme_ctrlr_submit_[admin|io]_request functions which consolidates
code for allocating nvme_tracker objects and making calls into
bus_dmamap_load for commands which have payloads.

Sponsored by:	Intel
2012-10-18 00:39:29 +00:00