- Do not explicitly count active descriptors. It allows hardware reset
to happen while device is still referenced, plus simplifies locking.
- Do not stop/start callout each time the queue becomes empty. Let it
run to completion and rearm if needed, that is much cheaper then to touch
it every time, plus also simplifies locking.
- Decouple submit and cleanup locks, making driver reentrant.
- Avoid memory mapped status register read on every interrupt.
- Improve locking during device attach/detach.
- Remove some no longer used variables.
Reviewed by: cem
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D19231
Even M_WAITOK callers must check for failure. For example, if the device is
quiescing, either due to automatic error-recovery induced reset, or due to
administrative detach, the routine will return ENXIO and the acquire
reference will not be held. So, there is no mode in which it is safe to
assume the routine succeeds without checking.
Sponsored by: Dell EMC Isilon
Remove unused and easy to misuse PNP macro parameter
Inspired by r338025, just remove the element size parameter to the
MODULE_PNP_INFO macro entirely. The 'table' parameter is now required to
have correct pointer (or array) type. Since all invocations of the macro
already had this property and the emitted PNP data continues to include the
element size, there is no functional change.
Mostly done with the coccinelle 'spatch' tool:
$ cat modpnpsize0.cocci
@normaltables@
identifier b,c;
expression a,d,e;
declarer MODULE_PNP_INFO;
@@
MODULE_PNP_INFO(a,b,c,d,
-sizeof(d[0]),
e);
@singletons@
identifier b,c,d;
expression a;
declarer MODULE_PNP_INFO;
@@
MODULE_PNP_INFO(a,b,c,&d,
-sizeof(d),
1);
$ rg -l MODULE_PNP_INFO -- sys | \
xargs spatch --in-place --sp-file modpnpsize0.cocci
(Note that coccinelle invokes diff(1) via a PATH search and expects diff to
tolerate the -B flag, which BSD diff does not. So I had to link gdiff into
PATH as diff to use spatch.)
Tinderbox'd (-DMAKE_JUST_KERNELS).
Approved by: re (glen)
I was not aware Warner was making or planning to make forward progress in
this area and have since been informed of that.
It's easy to apply/reapply when churn dies down.
Inspired by r338025, just remove the element size parameter to the
MODULE_PNP_INFO macro entirely. The 'table' parameter is now required to
have correct pointer (or array) type. Since all invocations of the macro
already had this property and the emitted PNP data continues to include the
element size, there is no functional change.
Mostly done with the coccinelle 'spatch' tool:
$ cat modpnpsize0.cocci
@normaltables@
identifier b,c;
expression a,d,e;
declarer MODULE_PNP_INFO;
@@
MODULE_PNP_INFO(a,b,c,d,
-sizeof(d[0]),
e);
@singletons@
identifier b,c,d;
expression a;
declarer MODULE_PNP_INFO;
@@
MODULE_PNP_INFO(a,b,c,&d,
-sizeof(d),
1);
$ rg -l MODULE_PNP_INFO -- sys | \
xargs spatch --in-place --sp-file modpnpsize0.cocci
(Note that coccinelle invokes diff(1) via a PATH search and expects diff to
tolerate the -B flag, which BSD diff does not. So I had to link gdiff into
PATH as diff to use spatch.)
Tinderbox'd (-DMAKE_JUST_KERNELS).
An eventual devd(8) or other component should be able to scan buses and
automatically load drivers that match device ids described in this metadata.
Reviewed by: imp
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12364
Modern GCC and Clang simply ignore the qualifier, while the old base GCC
produces a warning (treated as an error in the kernel build).
Approved by: cem
MFC after: 5 days
Fix ioat_release to only set is_completion_pending if DMAs were actually
queued. Otherwise, the spurious flag could trigger an assert in the
reset path on INVARIANTS kernels.
Reviewed by: bdrewery, Suraj Raju @ Isilon
Sponsored by: Dell EMC Isilon
In the case where a hardware error is detected during
ioat_process_events, hardware may advance (by one descriptor, probably)
and a subsequent ioat_process_events may race the intended ioat_reset_hw
followup. In that case, the second process_events would observe a
completion update that does not match the software "last_seen" status,
and attempt to successfully complete already-failed descriptors.
Guard against this race with the resetting_cleanup flag.
Reviewed by: bdrewery, markj
Sponsored by: Dell EMC Isilon
The device doesn't accurately update the CHANCMP address with the device state
when the device is suspended or halted. So, read the CHANSTS register to check
for those states.
We still need to read the CHANCMP address for the last completed descriptor.
Sponsored by: Dell EMC Isilon
This allows us to make strong assertions about descriptor address
validity. Additionally, future generations of the ioat(4) hardware will
require contiguous descriptors.
Reviewed by: markj
Sponsored by: Dell EMC Isilon
This paves the way for a contiguous descriptor array.
A contiguous descriptor array has the benefit that we can make strong
assertions about whether an address is a valid descriptor or not. The
other benefit is that future generations of I/OAT hardware will require
a contiguous descriptor array anyway. The downside is that after system
boot, big chunks of contiguous memory is much harder to find. So
dynamic scaling after boot is basically impossible.
Reviewed by: markj
Sponsored by: Dell EMC Isilon
The CHANSTS register is a split 64-bit register on CBDMA units before
hardware v3.3. If a torn read happens during ioat_process_events(),
software cannot know when to stop completing descriptors correctly.
So, just use the device-pushed main memory channel status instead.
Remove the ioat_get_active() seatbelt as well. It does nothing if the
completion address is valid.
Sponsored by: Dell EMC Isilon
In r304602, I mistakenly removed the ioat_process_events check that we weren't
processing events before the hardware had completed the descriptor
("last_seen"). Reinstate that logic.
Keep the defensive loop condition and additionally make sure we've actually
completed a descriptor before blindly chasing the ring around.
In reset, queue and finish the startup command before allowing any event
processing or submission to occur. Avoid potential missed callouts by
requeueing the poll later.
is_completion_pending governs whether or not a callout will be scheduled
when new work is queued on the IOAT device. If true, a callout is
already scheduled, so we do not need a new one. If false, we schedule
one and set it true. Because resetting the hardware completed all
outstanding work but failed to clear is_completion_pending, no new
callout could be scheduled after a reset with pending work.
This resulted in a driver hang for polled-only work.
Fix the race between ioat_reset_hw and ioat_process_events.
HW reset isn't protected by a lock because it can sleep for a long time
(40.1 ms). This resulted in a race where we would process bogus parts
of the descriptor ring as if it had completed. This looked like
duplicate completions on old events, if your ring had looped at least
once.
Block callout and interrupt work while reset runs so the completion end
of things does not observe indeterminate state and process invalid parts
of the ring.
Start the channel with a manually implemented ioat_null() to keep other
submitters quiesced while we wait for the channel to start (100 us).
r295605 may have made the race between ioat_reset_hw and
ioat_process_events wider, but I believe it already existed before that
revision. ioat_process_events can be invoked by two asynchronous
sources: callout (softclock) and device interrupt. Those could race
each other, to the same effect.
Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097
Add CRC/MOVECRC operations, as well as the TEST and STORE variants.
With these operations, a CRC32C can be computed over one or more
descriptors' source data. When the STORE operation is encountered, the
accumulated CRC32C is emitted to memory. A TEST operations triggers an
IOAT channel error if the accumulated CRC32C does not match one in
memory.
These operations are not exposed through any API yet.
Sponsored by: EMC / Isilon Storage Division
The IOAT engine can only address the low 40 bits (1 TB) of physmem via
the 'next descriptor' pointer. Restrict acceptable range given to
bus_dma_tag_create to match.
Sponsored by: EMC / Isilon Storage Division
