- Deduce allowed address range for bus_dma(9) from the hardware version.
Different versions (CPU generations) have different documented limits.
- Remove difference between address ranges for src/dst and crc. At least
docs for few recent generations of CPUs do not mention anything like that,
while older are already limited with above limits.
- Remove address assertions from arguments. While I do not think the
addresses out of allowed ranges should realistically happen there due to
the platforms physical address limitations, there is now bus_dma(9) to
make sure of that, preferably via IOMMU.
- Since crc now has the same address range as src/dst, remove crc_dmamap,
reusing dst2_dmamap instead.
Discussed with: cem
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
Allocate ioat->ring memory from the device domain.
Schedule ioat->poll_timer to the first CPU of the device domain.
According to pcm-numa tool from intel-pcm port, this reduces number of
remote DRAM accesses while copying data by 75%. And unless it is a noise,
I've noticed some speed improvement when copying data to other domain.
MFC after: 1 week
Sponsored by: iXsystems, Inc.
- Do not explicitly count active descriptors. It allows hardware reset
to happen while device is still referenced, plus simplifies locking.
- Do not stop/start callout each time the queue becomes empty. Let it
run to completion and rearm if needed, that is much cheaper then to touch
it every time, plus also simplifies locking.
- Decouple submit and cleanup locks, making driver reentrant.
- Avoid memory mapped status register read on every interrupt.
- Improve locking during device attach/detach.
- Remove some no longer used variables.
Reviewed by: cem
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D19231
Fix ioat_release to only set is_completion_pending if DMAs were actually
queued. Otherwise, the spurious flag could trigger an assert in the
reset path on INVARIANTS kernels.
Reviewed by: bdrewery, Suraj Raju @ Isilon
Sponsored by: Dell EMC Isilon
This allows us to make strong assertions about descriptor address
validity. Additionally, future generations of the ioat(4) hardware will
require contiguous descriptors.
Reviewed by: markj
Sponsored by: Dell EMC Isilon
This paves the way for a contiguous descriptor array.
A contiguous descriptor array has the benefit that we can make strong
assertions about whether an address is a valid descriptor or not. The
other benefit is that future generations of I/OAT hardware will require
a contiguous descriptor array anyway. The downside is that after system
boot, big chunks of contiguous memory is much harder to find. So
dynamic scaling after boot is basically impossible.
Reviewed by: markj
Sponsored by: Dell EMC Isilon
The CHANSTS register is a split 64-bit register on CBDMA units before
hardware v3.3. If a torn read happens during ioat_process_events(),
software cannot know when to stop completing descriptors correctly.
So, just use the device-pushed main memory channel status instead.
Remove the ioat_get_active() seatbelt as well. It does nothing if the
completion address is valid.
Sponsored by: Dell EMC Isilon
Fix the race between ioat_reset_hw and ioat_process_events.
HW reset isn't protected by a lock because it can sleep for a long time
(40.1 ms). This resulted in a race where we would process bogus parts
of the descriptor ring as if it had completed. This looked like
duplicate completions on old events, if your ring had looped at least
once.
Block callout and interrupt work while reset runs so the completion end
of things does not observe indeterminate state and process invalid parts
of the ring.
Start the channel with a manually implemented ioat_null() to keep other
submitters quiesced while we wait for the channel to start (100 us).
r295605 may have made the race between ioat_reset_hw and
ioat_process_events wider, but I believe it already existed before that
revision. ioat_process_events can be invoked by two asynchronous
sources: callout (softclock) and device interrupt. Those could race
each other, to the same effect.
Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097
Add CRC/MOVECRC operations, as well as the TEST and STORE variants.
With these operations, a CRC32C can be computed over one or more
descriptors' source data. When the STORE operation is encountered, the
accumulated CRC32C is emitted to memory. A TEST operations triggers an
IOAT channel error if the accumulated CRC32C does not match one in
memory.
These operations are not exposed through any API yet.
Sponsored by: EMC / Isilon Storage Division
The I/OAT HW reset process may sleep, so it is invalid to perform a
channel reset from the software interrupt thread.
Sponsored by: EMC / Isilon Storage Division
In I/OAT, this is done through the INTRDELAY register. On supported
platforms, this register can coalesce interrupts in a set period to
avoid excessive interrupt load for small descriptor workflows. The
period is configurable anywhere from 1 microsecond to 16.38
milliseconds, in microsecond granularity.
Sponsored by: EMC / Isilon Storage Division
The hardware supports descriptors with two non-contiguous pages. This
allows issuing one descriptor for an 8k copy from/to non-contiguous but
otherwise page-aligned memory.
Sponsored by: EMC / Isilon Storage Division
Certain invalid operations trigger hardware error conditions. Error
conditions that only halt one channel can be detected and recovered by
resetting the channel. Error conditions that halt the whole device are
generally not recoverable.
Add a sysctl to inject channel-fatal HW errors,
'dev.ioat.<N>.force_hw_error=1'.
When a halt due to a channel error is detected, ioat(4) blocks new
operations from being queued on the channel, completes any outstanding
operations with an error status, and resets the channel before allowing
new operations to be queued again.
Update ioat.4 to document error recovery; document blockfill introduced
in r290021 while we are here; document ioat_put_dmaengine() added in
r289907; document DMA_NO_WAIT added in r289982.
Sponsored by: EMC / Isilon Storage Division
Add generic hw descriptor struct and generic control flags struct, in
preparation for other kinds of IOAT operation.
Sponsored by: EMC / Isilon Storage Division
Add a new flag for DMA operations, DMA_NO_WAIT. It behaves much like
other NOWAIT flags -- if queueing an operation would sleep, abort and
return NULL instead.
When growing the internal descriptor ring, the memory allocation is
performed outside of all locks. A lock-protected flag is used to avoid
duplicated work. Threads that cannot sleep and attempt to queue
operations when the descriptor ring is full allocate a larger ring with
M_NOWAIT, or bail if that fails.
ioat_reserve_space() could become an external API if is important to
callers that they have room for a sequence of operations, or that those
operations succeed each other directly in the hardware ring.
This patch splits the internal head index (->head) from the hardware's
head-of-chain (DMACOUNT) register (->hw_head). In the future, for
simplicity's sake, we could drop the 'ring' array entirely and just use
a linked list (with head and tail pointers rather than indices).
Suggested by: Witness
Sponsored by: EMC / Isilon Storage Division
We only need to borrow a mutex for the drain sleep and the 0->1
transition, so just reuse an existing one for now.
The wchan is arbitrary. Using refcount itself would have required
__DEVOLATILE(), so use the lock's address instead.
Different uses are tagged by kind, although we only do anything with
that information in INVARIANTS builds.
Sponsored by: EMC / Isilon Storage Division
Replace custom Linux-like logging with a thin shim around
device_printf(), when the softc is available.
In ioat_test, shim around printf(9) instead.
Sponsored by: EMC / Isilon Storage Division
Don't run the selftest until after we've enabled bus mastering, or the
DMA engine can't copy anything for our test.
Create the ioat_test device on attach, if so tuned. Destroy the
ioat_test device on teardown.
Replace deprecated 'CALLOUT_MPSAFE' with correct '1' in callout_init().
Sponsored by: EMC / Isilon Storage Division
I/OAT is also referred to as Crystal Beach DMA and is a Platform Storage
Extension (PSE) on some Intel server platforms.
This driver currently supports DMA descriptors only and is part of a
larger effort to upstream an interconnect between multiple systems using
the Non-Transparent Bridge (NTB) PSE.
For now, this driver is only built on AMD64 platforms. It may be ported
to work on i386 later, if that is desired. The hardware is exclusive to
x86.
Further documentation on ioat(4), including API documentation and usage,
can be found in the new manual page.
Bring in a test tool, ioatcontrol(8), in tools/tools/ioat. The test
tool is not hooked up to the build and is not intended for end users.
Submitted by: jimharris, Carl Delsey <carl.r.delsey@intel.com>
Reviewed by: jimharris (reviewed my changes)
Approved by: markj (mentor)
Relnotes: yes
Sponsored by: Intel
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D3456
