Commit Graph

184 Commits

Author SHA1 Message Date
Alexander Motin
3a76d901d6 Include sys/lock.h, as told by man page.
MFC after:	1 week
2019-07-02 15:01:54 +00:00
Alexander Motin
7b96ad44dd Fix i386 LINT after r349594.
MFC after:	1 month
2019-07-02 07:47:11 +00:00
Alexander Motin
6683132d54 Add driver for NTB in AMD SoC.
This patch is the driver for NTB hardware in AMD SoCs (ported from Linux)
and enables the NTB infrastructure like Doorbells, Scratchpads and Memory
window in AMD SoC. This driver has been validated using ntb_transport and
if_ntb driver already available in FreeBSD.

Submitted by:	Rajesh Kumar <rajesh1.kumar@amd.com>
MFC after:	1 month
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D18774
2019-07-02 05:25:18 +00:00
Conrad Meyer
e2e050c8ef Extract eventfilter declarations to sys/_eventfilter.h
This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.

EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).

As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions.  The remainder of the patch addresses
adding appropriate includes to fix those files.

LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).

No functional change (intended).  Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed.  __FreeBSD_version has been bumped.
2019-05-20 00:38:23 +00:00
Alexander Motin
a8bc5594db Allow I/OAT of present Xeon E5/E7 to work thorugh PLX NTB.
Its a hack, we can't know/list all DMA engines, but this covers all
I/OAT of Xeon E5/E7 at least from Sandy Bridge till Skylake I saw.

MFC after:	1 week
Sponsored by:	iXsystems, Inc.
2019-02-21 14:10:14 +00:00
Warner Losh
329e817fcc Reapply, with minor tweaks, r338025, from the original commit:
Remove unused and easy to misuse PNP macro parameter

Inspired by r338025, just remove the element size parameter to the
MODULE_PNP_INFO macro entirely.  The 'table' parameter is now required to
have correct pointer (or array) type.  Since all invocations of the macro
already had this property and the emitted PNP data continues to include the
element size, there is no functional change.

Mostly done with the coccinelle 'spatch' tool:

  $ cat modpnpsize0.cocci
    @normaltables@
    identifier b,c;
    expression a,d,e;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,d,
    -sizeof(d[0]),
     e);

    @singletons@
    identifier b,c,d;
    expression a;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,&d,
    -sizeof(d),
     1);

  $ rg -l MODULE_PNP_INFO -- sys | \
    xargs spatch --in-place --sp-file modpnpsize0.cocci

(Note that coccinelle invokes diff(1) via a PATH search and expects diff to
tolerate the -B flag, which BSD diff does not.  So I had to link gdiff into
PATH as diff to use spatch.)

Tinderbox'd (-DMAKE_JUST_KERNELS).
Approved by: re (glen)
2018-09-26 17:12:14 +00:00
Conrad Meyer
b8e771e97a Back out r338035 until Warner is finished churning GSoC PNP patches
I was not aware Warner was making or planning to make forward progress in
this area and have since been informed of that.

It's easy to apply/reapply when churn dies down.
2018-08-19 00:46:22 +00:00
Conrad Meyer
faa319436f Remove unused and easy to misuse PNP macro parameter
Inspired by r338025, just remove the element size parameter to the
MODULE_PNP_INFO macro entirely.  The 'table' parameter is now required to
have correct pointer (or array) type.  Since all invocations of the macro
already had this property and the emitted PNP data continues to include the
element size, there is no functional change.

Mostly done with the coccinelle 'spatch' tool:

  $ cat modpnpsize0.cocci
    @normaltables@
    identifier b,c;
    expression a,d,e;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,d,
    -sizeof(d[0]),
     e);

    @singletons@
    identifier b,c,d;
    expression a;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,&d,
    -sizeof(d),
     1);

  $ rg -l MODULE_PNP_INFO -- sys | \
    xargs spatch --in-place --sp-file modpnpsize0.cocci

(Note that coccinelle invokes diff(1) via a PATH search and expects diff to
tolerate the -B flag, which BSD diff does not.  So I had to link gdiff into
PATH as diff to use spatch.)

Tinderbox'd (-DMAKE_JUST_KERNELS).
2018-08-19 00:22:21 +00:00
Alexander Motin
6e76bc32d4 Try to preallocate receive memory early.
We may not have enough contiguous memory later, when NTB connection get
established.  It is quite likely that NTB windows are symmetric and this
allocation remain, but even if not, we will just reallocate it later.

MFC after:	2 weeks
2018-01-31 01:04:36 +00:00
Alexander Motin
3986af1f65 Use bus_dmamem_alloc(9) KPI instead of contigmalloc(9).
I suppose it should make this code NUMA-aware with recent NUMA drop-in,
trying to allocate shared memory buffers from domain closer to NT-bridge.

MFC after:	2 weeks
2018-01-26 22:22:26 +00:00
Warner Losh
6efa5583c7 Fix typos from last commit, these should have been #. 2017-12-22 20:48:49 +00:00
Warner Losh
d2064cf030 Use '#' rather than some made up name for fields we want to ignore. 2017-12-22 17:53:27 +00:00
Alexander Motin
21811c24c6 Add initial support for Address Lookup Table (A-LUT).
When enabled by EEPROM, use it to relax translation address/size alignment
requirements for BAR2 window by 128 or 256 times.

MFC after:	1 week
Sponsored by:	iXsystems, Inc.
2017-10-01 09:48:31 +00:00
Conrad Meyer
a64bf59c49 Add PNP metadata to a few drivers
An eventual devd(8) or other component should be able to scan buses and
automatically load drivers that match device ids described in this metadata.

Reviewed by:	imp
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12364
2017-09-14 15:34:45 +00:00
Alexander Motin
83feae78cc Add second entry to LUT on a link side in B2B mode.
Each of two entries on a virtual side should have its counterpart on a
peer's link side.

MFC after:	1 week
2017-09-14 04:51:17 +00:00
Alexander Motin
3dcd9d784a Increase negotiation polling period from 10ms to 100ms.
There is no big need to burn CPU if other side may be not there yet.  For
example, the PLX hardware by default enables the NTB link up on reset, not
dependig on driver to do it.  In case of Intel hardware this also reduces
race between MSI-X workaround negotiation and upper layers, using the same
scratchpad registers in different time.

MFC after:	12 days
2017-09-02 13:28:45 +00:00
Alexander Motin
b7dd3fbede Make NTB drivers report more info via NewBus methods.
MFC after:	12 days
2017-09-02 11:56:16 +00:00
Alexander Motin
08dc78166d Link Interface has no Link Error registers.
MFC after:	13 days
2017-09-01 09:48:19 +00:00
Alexander Motin
84f8cfec2f Clear doorbell bits after masking them before processing.
In theory this allows to avoid one more expensive doorbell register read
later in some scenarios.  But in practice it also significantly increases
packet rate on PLX hardware, that I can't explain yet, possibly work-
arounding some interrupt delays.

MFC after:	13 days
Sponsored by:	iXsystems, Inc.
2017-08-31 21:37:22 +00:00
Alexander Motin
171e3d1a1f Remove unneeded pmap_change_attr() calls.
Reported by:	kib
MFC after:	13 days
2017-08-31 17:02:06 +00:00
Alexander Motin
0faf59149c Add/polish some defines.
MFC after:	13 days
2017-08-31 16:32:11 +00:00
Alexander Motin
b6c46372af Fix port control for PEX 8749.
That chip has three Station Ports, so previous address math was incorrect.

MFC after:	13 days
Sponsored by:	iXsystems, Inc.
2017-08-31 13:41:44 +00:00
Alexander Motin
c7dabb6563 Make ntb_set_ctx() always generate fake link event.
It allows application driver get initial link state without racing with
hardware interrupts, thanks to the context rmlock held here.

MFC after:	1 week
Sponsored by:	iXsystems, Inc.
2017-08-31 10:59:39 +00:00
Alexander Motin
ba4b25cbba Make ntb_transport(4) ready receive early link events.
Those events may be reported as soon as callback is registered, if the link
is enabled by hardware or some other application.

While there, clean link_is_up variable on link down event.

MFC after:	1 week
2017-08-31 10:53:10 +00:00
Alexander Motin
ed9652da5f Add NTB driver for PLX/Avago/Broadcom PCIe switches.
This driver supports both NTB-to-NTB and NTB-to-Root Port modes (though
the second with predictable complications on hot-plug and reboot events).
I tested it with PEX 8717 and PEX 8733 chips, but expect it should work
with many other compatible ones too.  It supports up to two NT bridges
per chip, each of which can have up to 2 64-bit or 4 32-bit memory windows,
6 or 12 scratchpad registers and 16 doorbells.  There are also 4 DMA engines
in those chips, but they are not yet supported.

While there, rename Intel NTB driver from generic ntb_hw(4) to more specific
ntb_hw_intel(4), so now it is on par with this new ntb_hw_plx(4) driver and
alike to Linux naming.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2017-08-30 21:16:32 +00:00
Alexander Motin
546ec4e544 Mask doorbells while processing them.
This fixes interrupt storms on hardware using legacy level-triggered
interrupts, since doorbell processing could take time after interrupt
handler completion, that triggered extra interrupts in a loop.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2017-08-28 20:00:21 +00:00
Alexander Motin
e6f000753e Fix fake interrupt when set doorbell is unmasked.
Since the doorbell bit is already set when interrupt handler is called,
the event was not propagated to upper layer.  It was working normally
because present code was not using masking actively, but that is going
to change.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2017-08-28 19:52:57 +00:00
Alexander Motin
48c4767756 Report NTB link speed to console and interface.
MFC after:	2 weeks
2017-04-23 14:25:51 +00:00
Alexander Motin
47967ab741 Pretend we support some IOCTLs to not scary upper layers.
MFC after:	2 weeks
2017-01-11 20:23:45 +00:00
Alexander Motin
7e58465356 Wrap previous MSIX workaround into #ifndef EARLY_AP_STARTUP.
With EARLY_AP_STARTUP we can successfully negotiate MSIX earlier.

Requested by:	jhb@
2016-07-30 21:06:59 +00:00
Alexander Motin
a8ec75016f Block MSIX negotiation until SMP started and IRQ reshuffled. 2016-07-30 15:56:36 +00:00
Alexander Motin
8260941c76 Make MAC address generation more random.
'ticks' approach does not work at boot time.
2016-07-30 15:51:16 +00:00
Alexander Motin
cf2e151f65 Fix infinite loops introduced at r303429. 2016-07-30 10:32:28 +00:00
Alexander Motin
718e8abba9 Fix NTBT_QP_LINKS negotiation.
I believe it never worked correctly for more the one queue even in Linux.
This fixes case when one of consumer drivers is not loaded on one side,
but its queues still announced as ready if something else brought link up.

While there, remove some pointless NULL checks.
2016-07-29 21:03:30 +00:00
Alexander Motin
75b94efd08 Clear scratchpad after MSIX negotiation to not leak garbage. 2016-07-29 20:52:18 +00:00
Alexander Motin
6bd57d14ed Once more refactor KPI between ntb_transport(4) and if_ntb(4)..
New design allows to attach multiple consumers to ntb_transport(4) instance.
Previous design obtained from Linux theoretically allowed that, but was not
practically usable (Linux also has only one consumer driver now).
2016-07-29 17:15:41 +00:00
Alexander Motin
8709930cfc Fix r303429 build with invariants. 2016-07-28 12:25:58 +00:00
Alexander Motin
4490696b3e Once more refactor KPI between NTB hardware and consumers.
New design allows hardware resources to be split between several consumers.
For example, one BAR can be dedicated for remote memory access, while other
resources can be used for packet transport for virtual Ethernet interface.
And even without resource split, this code allows to specify which consumer
driver should attach the hardware.

From some points this makes the code even closer to Linux one, even though
Linux does not provide the described flexibility.
2016-07-28 10:48:20 +00:00
Alexander Motin
c072301617 Postpone ntb_get_msix_info() till we need to negotiate MSIX.
Calling it earlier increases the window when MSIX info may change.
This change does not solve the problem completely, but seems logical.
Complete solution should probably include link reset in case of MSIX
remap to trigger new negotiation, but we have no way to get notified
about that now.
2016-07-24 14:42:11 +00:00
Sepherosa Ziehau
02888b1e1b ntb: Fix LINT
Sponsored by:	Microsoft OSTC
2016-07-12 05:41:34 +00:00
Alexander Motin
ce35eeb771 Revert odd change, setting limit registers before base.
I don't know what errata is mentioned there, I was unable to find it, but
setting limit before the base simply does not work at all.  According to
specification attempt to set limit out of the present window range resets
it to zero, effectively disabling it.  And that is what I see in practice.

Fixing this properly disables access for remote side to our memory until
respective xlat is negotiated and set.  As I see, Linux does the same.
2016-07-10 20:22:04 +00:00
Alexander Motin
9c85a81906 Fix wrong copy/paste in r302510. 2016-07-10 19:52:26 +00:00
Alexander Motin
d2de6786c1 Remove callout_reset(link_work) from ntb_transport_attach().
At that point link is quite likely not established yet, so messing with
scratch registers is premature there.  Original commit message mentioned
code diff reduction from Linux, but this line is not present in Linux now.
2016-07-10 19:15:29 +00:00
Alexander Motin
a8916479cc Simplify MSIX MW BAR xlat setup, and don't forget to unlock its limit.
The last fixes SB01BASE_LOCKUP workaround after driver reload.
2016-07-10 01:09:16 +00:00
Alexander Motin
eeed92a759 Disable SB01BASE_LOCKUP workaround when split BARs disabled.
For some reason hack with sending MSI-X interrupts by writing to remote
LAPIC memory works only for 32-bit BARs, that are available only if split
BARs mode is enabled in BIOS.  If it is not, complain loudly and fall back
to less efficient workaround.
2016-07-09 23:22:44 +00:00
Alexander Motin
c266ab3830 Improve checksum "offload" support.
For compatibility reasons make driver not report any checksum offload by
default, since there is indeed none.  But if administrator knows that
interface is used only for local traffic, he can enable fake checksum
offload manually on both sides to save some CPU cycles, since the data
are already protected by CRC32 of PCIe link.

Sponsored by:	iXsystems, Inc.
2016-07-09 14:59:16 +00:00
Alexander Motin
8795de77ce Rewrite if_ntb to use modern interface KPIs and features.
It includes: link state, if_transmit, buf_ring, multiple queues, bpf, etc.

Sponsored by:	iXsystems, Inc.
2016-07-09 12:10:08 +00:00
Alexander Motin
b42e3ec60b Improve memory allocation errors handling on receive. 2016-07-09 12:00:04 +00:00
Alexander Motin
a7e0f01a3d Synchronize MTU code with Linux.
It is mandatory for transport compatibility.
2016-07-09 11:58:59 +00:00
Alexander Motin
59850913b2 Reimplement doorbell register emulation for NTB_SB01BASE_LOCKUP.
This allows at least first three doorbells to work very close to normal
hardware, properly signaling events to upper layers without spurious or
lost events.  Doorbells above the first three may still report spurious
events due to lack of reliable information, but they are rarely used.
2016-07-09 11:57:21 +00:00