This is a holdover from how reset is handled in the ARGE_MDIO world.
You need to define the mdio bus device if you want to use the ethernet
device or the arge setup path doesn't bring the MAC out of reset.
The QCA953x SoC is an integrated 2x2 2GHz 11n + MIPS24k core, with
a 5 port FE switch, gige WAN port, and all the same stuff you'd find on
its predecessor - the AR9331.
However, buried deep in here somewhere is also a PCIe EP/RC for various
applications and some other weird bits I don't yet know about.
This is enough to get the reference board up and booting. I haven't yet
had it pass lots of packets - I need to finalise the ethernet switch
bits and the GMAC configuration (ie, how the ethernet ports and switch
are wired up) and I'll bring that in when I commit the base configuration
files to use the thing.
The wifi stuff will come much later. I have to port that support from
Linux ath9k and extend our vendor HAL to support it.
The reference board (AP143) comes with 32MB RAM and 4MB flash, so in order
to use it I need to get USB working fully so I can run root from there.
Thankyou to Qualcomm Atheros for access to the reference design board.
Details:
* Add register definitions from openwrt;
* It looks like a QCA955x but shrunk down to a QCA933x footprint, so
use the QCA955x bits and fix up the clock detection code to do the
QCA953x bits (they're very subtly different);
* Teach GPIO about it;
* Teach EHCI about it;
* Teach if_arge about it;
* Teach the CPU detection code about it.
Tested:
* AP143, QCA9533v2 SoC
Obtained from: Linux, Linux OpenWRT
* use barriers in a slightly better fashion. You can blame this
glass of whiskey on putting barriers in the wrong spot. Grr adrian.
* steal/rewrite the mdio busy check from ag7100 from openwrt and
refactor the existing code out. This is .. more correct.
This seems to fix the boot-to-boot variation that I've been seeing
and it quietens the switch port status flapping.
Tested:
* QCA9558 SoC (AP135.)
Obtained from: Linux OpenWRT
This driver and the linux ag71xx driver both treat the transmit ring
as a circular linked list of descriptors. There's no "end" pointer
that is ever NULL - instead, it expects the MAC to hit a finished
descriptor (ARGE_DESC_EMPTY) and stop.
Now, since it's a circular buffer, we may end up with the hardware
hitting the beginning of our multi-descriptor frame before we've finished
setting it up. It then DMA's it in, starts sending it, and we finish
writing out the new descriptor. The hardware may then write its
completion for the next descriptor out; then we do, and when we next
read it it'll show up as "not done" and transmit completion stops.
This unfortunately manifests itself as the transmit queue always
being active and a massive TX interrupt storm. We need to actively
ACK packets back from the transmit engine and if we don't (eg because
we think the transmit isn't finished but it is) then the unit will
just keep generating interrupts.
I hit this finally with the below testing setup. This fixed it for me.
Strictly speaking I should put in a sync in between writing out all of
the descriptors and writing out that final descriptor.
Tested:
* QCA9558 SoC (AP135 reference board) w/ arge1 + vlans acting as a
router, and iperf -d (tcp, bidirectional traffic.)
Obtained from: Linux OpenWRT (ag71xx_main.c.)
The MIPS busdma sync operations currently are a big no-op on coherent memory.
This isn't strictly correct behaviour as we need a SYNC in here to ensure that
the writes have finished and are visible in main memory before the MMIO accesses
occur. This will have to be addressed in a later commit.
But, before that happens, let's at least do a flush here to make things
more "correct".
This is required for even remotely sensible behaviour on mips74k with
write-through memory enabled.
The mips74k programmers guide notes that reads can be re-ordered, even
uncached ones, so we need an explicit SYNC between them.
Yes, this is a case of a driver author actively doing a bus barrier
operation.
This ends up being necessary when the mips74k core is run in write-back
mode rather than write-through mode. That's coming in an upcoming
commit.
Tested:
* mips74k, QCA9558 SoC (AP135 reference board), arge<->arge interface
routing traffic tests.
send frames.
This matches the other check for space.
"enough" is a misnomer, for "reasons". The biggest reason is that
the TX ring is actually a circular linked list, with no head/tail pointers.
This is just a bit more headroom between head/tail so we have time to
schedule frames before we hit where the hardware is at.
Ideally this would be tunable and a little larger.
This flushes out the write to the system before anything continues.
The mips74k guide, chapter 3.3.3 (write gathering) notes that writes
can be buffered in FIFOs - even uncached ones - so we can't guarantee
the device has felt its effects. Now, since we're all lazy driver
authors and don't pepper read/write barriers everywhere, fake it here.
tested:
* mips74k - QCA9558 SoC (AP135 reference board)
This should make it easier to track down interrupt storms from arge.
Tested:
* AP135 (QCA955x) SoC - defaults to ARGE_DEBUG enabled
* Carambola2 (AR9331 SoC) - defaults to ARGE_DEBUG disabled
I couldn't test arge0->arge1 bridging, only arge0 VLAN bridging.
The DIR-825C1 only hooks up arge0 to the switch GMAC0 and so
you need to abuse VLANs to test.
Tested:
* DIR-825C1 (AR9344)
This part seems to work bug-free with single byte TX/RX buffer alignment.
This drops the CPU requirement to bridge 100mbit iperf from 100% CPU
to ~ 50% CPU.
Tested:
* AP121 (AR9330) SoC, highly magic netbooted kernel + USB rootfs
due to 4mb flash, 16mb RAM; doing bridging between arge0 and arge1.
Notes:
* Yes, I likely can also turn this on for the AR934x SoC family now.
But since hardware design apparently follows similar branching
strategies to software design, I'll go and make sure all the AR934x's
that made it out into shipping products work before I flip it on.
This was triggering when using it as an AP bridge rather than an ethernet
bridge.
The code is unclear but it works; I'll fix it to be clearer and test
performance at a later stage.
The existing code meets the "alignment" requirement for the l3 payload
by offsetting the mbuf by uint64_t and then calling an rx fixup routine
to copy the frame backwards by 2 bytes. This DWORD aligns the
L3 payload so tcp, etc doesn't panic on unaligned access.
This is .. slow.
For arge MACs that support 1 byte TX/RX address alignment, we can do
the "other" hack: offset the RX address of the mbuf so the L3 payload
again is hopefully DWORD aligned.
This is much cheaper - since TX/RX is both 1 byte align ready (thanks
to the previous commit) there's no bounce buffering going on and there
is no rx fixup copying.
This gets bridging performance up from 180mbit/sec -> 410mbit/sec.
There's around 10% of CPU cycles spent in _bus_dmamap_sync(); I'll
investigate that later.
Tested:
* QCA955x SoC (AP135 reference board), bridging arge0/arge1
by programming the switch to have two vlangroups in dot1q mode:
# ifconfig bridge0 inet 192.168.2.20/24
# etherswitchcfg config vlan_mode dot1q
# etherswitchcfg vlangroup0 members 0,1,2,3,4
# etherswitchcfg vlangroup1 vlan 2 members 5,6
# etherswitchcfg port5 pvid 2
# etherswitchcfg port6 pvid 2
# ifconfig arge1 up
# ifconfig bridge0 addm arge1
The early ethernet MACs (I think AR71xx and AR913x) require that both
TX and RX require 4-byte alignment for all packets.
The later MACs have started relaxing the requirements.
For now, the 1-byte TX and 1-byte RX alignment requirements are only for
the QCA955x SoCs. I'll add in the relaxed requirements as I review the
datasheets and do testing.
* Add a hardware flags field and 1-byte / 4-byte TX/RX alignment.
* .. defaulting to 4-byte TX and 4-byte RX alignment.
* Only enforce the TX alignment fixup if the hardware requires a 4-byte
TX alignment. This avoids a call to m_defrag().
* Add counters for various situations for further debugging.
* Set the 1-byte and 4-byte busdma alignment requirement when
the tag is created.
This improves the straight bridging performance from 130mbit/sec
to 180mbit/sec, purely by removing the need for TX path bounce buffers.
The main performance issue is the RX alignment requirement and any RX
bounce buffering that's occuring. (In a local test, removing the RX
fixup path and just aligning buffers raises the performance to above
400mbit/sec.
In theory it's a no-op for SoCs before the QCA955x.
Tested:
* QCA9558 SoC in AP135 board, using software bridging between arge0/arge1.
When the system has more than a single PCI domain, the bus numbers
are not unique, thus they cannot be used for "pci" device numbering.
Change bus numbers to -1 (i.e. to-be-determined automatically)
wherever the code did not care about domains.
Reviewed by: jhb
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3406
and start teaching subsystems about it.
The Atheros MIPS platforms don't guarantee any kind of FIFO consistency
with interrupts in hardware. So software needs to do a flush when it
receives an interrupt and before it calls the interrupt handler.
There are new ones for the QCA934x and QCA955x, so do a few things:
* Get rid of the individual ones (for ethernet and IP2);
* Create a mux and enum listing all the variations on DDR flushes;
* replace the uses of IP2 with the relevant one (which will typically
be "PCI" here);
* call the USB DDR flush before calling the real USB interrupt handlers;
* call the ethernet one upon receiving an interrupt that's for us,
rather than never calling it during operation.
Tested:
* QCA9558 (TP-Link archer c7 v2)
* AR9331 (Carambola 2)
TODO:
* PCI, USB, ethernet, etc need to do a double-check to see if the
interrupt was truely for them before doing the DDR. For now I
prefer "correct" over "fast".
The QCA955x looks a lot like the AR724x PCIe controller, except it
supports two root complexes. Unfortunately I only have one, so
although this code has started down the path of supporting more than
one, it's definitely not yet ready.
Tested:
* AP135 board (QCA9558 SoC), with the 11ac NIC swapped for an AR9380
PCIe NIC.
Notes:
* Yes, this driver isn't very pretty. I decided to commit what I have
versus holding onto something that isn't yet finished. It is enough
to bring up the above NIC and interrupt routing works, so it's a good
start.
* However, yes, the DDR flush routine hooks need to be fixed up.
I don't think I'm firing the right one at the moment.
This is needed with the pl011 driver. Before this change it would default
to a shift of 0, however the hardware places the registers at 4-byte
addresses meaning the value should be 2.
This patch fixes this for the pl011 when configured using the fdt. The
other drivers have a default value of 0 to keep this a no-op.
MFC after: 1 week
A lot of these dinky atheros based MIPS boards don't have a nice, well,
anything consistent defining their MAC addresses for things.
The Atheros reference design boards will happily put MAC addresses
into the wifi module calibration data like they should, and individual
ethernet MAC addresses into the calibration area in flash.
That makes my life easy - "hint.arge.X.eeprommac=<addr>" reads from
that flash address to extract a MAC, and everything works fine.
However, aside from some very well behaved vendors (eg the Carambola 2
board), everyone else does something odd.
eg:
* a MAC address in the environment (eg ubiquiti routerstation/RSPRO)
that you derive arge0/arge1 MAC addresses from.
* a MAC address in flash that you derive arge0/arge1 MAC addresses from.
* The wifi devices having their own MAC addresses in calibration data,
like normal.
* The wifi devices having a fixed, default or garbage value for a MAC
address in calibration data, and it has to be derived from the
system MAC.
So to support this complete nonsense of a situation, there needs to be
a few hacks:
* The "board" MAC address needs to be derived from somewhere and squirreled
away. For now it's either redboot or a MAC address stored in calibration
flash.
* Then, a "map" set of hints to populate kenv with some MAC addresses
that are derived/local, based on the board address. Each board has
a totally different idea of what you do to derive things, so each
map entry has an "offset" (+ve or -ve) that's added to the board
MAC address.
* Then if_arge (and later, if_ath) should check kenv for said hint and
if it's found, use that rather than the EEPROM MAC address - which may
be totally garbage and not actually work right.
In order to do this, I've undone some of the custom redboot expecting
hacks in if_arge and the stuff that magically adds one to the MAC
address supplied by the board - instead, as I continue to test this
out on more hardware, I'll update the hints file with a map explaining
(a) where the board MAC should come from, and (b) what offsets to use
for each device.
The aim is to have all of the tplink, dlink and other random hardware
we run on have valid MAC addresses at boot, so (a) people don't get
random B:S:D❌x:x ethernet MACs, and (b) the wifi MAC is valid
so it works rather than trying to use an invalid address that
actually upsets systems (think: multicast bit set in BSSID.)
Tested:
* TP-Link TL_WDR3600 - subsequent commits will add the hints map
and the if_ath support.
TODO:
* Since this is -HEAD, and I'm all for debugging, there's a lot of
printf()s in here. They'll eventually go under bootverbose.
* I'd like to turn the macaddr routines into something available
to all drivers - too many places hand-roll random MAC addresses
and parser stuff. I'd rather it just be shared code.
However, that'll require more formal review.
* More boards.
The AR934x (and maybe others in this family) have a more complicated
GPIO mux. The AR71xx just has a single function register for a handful
of "GPIO or X" options, however the AR934x allows for one of roughly
100 behaviours for each GPIO pin.
So, this adds a quick hints based mechanism to configure the output
functions, which is required for some of the more interesting board
configurations. Specifically, some use external LNAs to improve
RX, and without the MUX/output configured right, the 2GHz RX side
will be plain terrible.
It doesn't yet configure the "input" side yet; I'll add that if
it's required.
Tested:
* TP-Link TL-WDR3600, testing 2GHz STA/AP modes, checking some
basic RX sensitivity things (ie, "can I see the AP on the other
side of the apartment that intentionally has poor signal reception
from where I am right now.")
Whilst here, fix a silly bug in the maxpin routine; I was missing
a break.
A lot of these embedded boards don't have a unique MAC address per
device stored somewhere unique - sometimes they'll have one MAC
for both arge NICs; someties they'll have one MAC for both arge NICs
/and/ the ath NICs. In these instances, we need to derive device
specific MAC addresses from the base MAC address.
These functions will be used by some follow-up code that'll slot
into if_arge and if_ath.
Otherwise, the initial media speed would change if a PHY is hooked up,
sending PHY speed notifications. For the AP135 at least, the RGMII
PHY has a static speed/duplex configured and if the PHY plumbing
attaches the PHY to the if_arge interface, the first link speed change
from 1000/full will set the MAC to something that isn't useful.
This shouldn't affect any other platforms - everything I looked at is
using hard-coded speed/duplex as static, as they're facing a switch
with no PHY attached.
There's two EHCI controllers in the QCA955x SoCs - they have different
interrupts available via various demux registers, but they both tie to
IP3.
So for now, allow them to be sharable so they can hang off of IP3.
There's a lot more to come - the QCA955x has a bunch more GPIO MUX
configuration, reminiscent of what the ARM chips let you do - but
it'll have to come later.
