events.
This is primarily for the TX EDMA and TX EDMA completion. I haven't yet
tied it into the EDMA RX path or the legacy TX/RX path.
Things that I don't quite like:
* Make the pointer type 'void' in ath_softc and have if_ath_alq*()
return a malloc'ed buffer. That would remove the need to include
if_ath_alq.h in if_athvar.h.
* The sysctl setup needs to be cleaned up.
I've tried serialising TX using queues and such but unfortunately
due to how this interacts with the locking going on elsewhere in the
networking stack, the TX task gets delayed, resulting in quite a
noticable throughput loss:
* baseline TCP for 2x2 11n HT40 is ~ 170mbit/sec;
* TCP for TX task in the ath taskq, with the RX also going on - 80mbit/sec;
* TCP for TX task in a separate, second taskq - 100mbit/sec.
So for now I'm going with the Linux wireless stack approach - lock tx
early. The linux code does in the wireless stack, before the 802.11
state stuff happens and before it's punted to the driver.
But TX locking needs to also occur at the driver layer as the TX
completion code _also_ begins to drain the ifnet TX queue.
Whilst I'm here, add some KTR traces for the TX path.
Note:
* This really should be done at the net80211 layer (as well, at least.)
But that'll have to wait for a little more thought to happen.
the power save queue.
* introduce some new ATH_NODE lock protected fields, tracking the
net80211 psq and TIM state;
* when doing buffer transitions - ie, when sending and completing
buffers - check the state of the SWQ and update the TIM appropriately.
* when clearing the TIM bit, if the SWQ is not empty then delay clearing
it.
This is racy, but it's no less racy than the current net80211 power
save queue management code. Specifically, with multiple TX threads,
it's quite plausible that parallel state updates will race and the
TIM will be left in an inconsistent state. I'll address that in
a follow-up commit.
stashed away in ath_node.
As much as I tried to stuff that behind the ATH_NODE lock, unfortunately
the locking is just too plain hairy (for me! And I wrote it!) to do
cleanly. Hence using atomics here instead of a lock. The ATH_NODE lock
just isn't currently used anywhere besides the rate control updates.
If in the future everything gets migrated back to using a single ATH_NODE
lock or a single global ATH_TX lock (ie, a single TX lock for all TX and
TX completion) then fine, I'll remove the atomics.
it run out of multiple concurrent contexts.
Right now the ath(4) TX processing is a bit hairy. Specifically:
* It was running out of ath_start(), which could occur from multiple
concurrent sending processes (as if_start() can be started from multiple
sending threads nowdays.. sigh)
* during RX if fast frames are enabled (so not really at the moment, not
until I fix this particular feature again..)
* during ath_reset() - so anything which calls that
* during ath_tx_proc*() in the ath taskqueue - ie, TX is attempted again
after TX completion, as there's now hopefully some ath_bufs available.
* Then, the ic_raw_xmit() method can queue raw frames for transmission
at any time, from any net80211 TX context. Ew.
This has caused packet ordering issues in the past - specifically,
there's absolutely no guarantee that preemption won't occuring _during_
ath_start() by the TX completion processing, which will call ath_start()
again. It's a mess - 802.11 really, really wants things to be in
sequence or things go all kinds of loopy.
So:
* create a new task struct for TX'ing;
* make the if_start method simply queue the task on the ath taskqueue;
* make ath_start() just be called by the new TX task;
* make ath_tx_kick() just schedule the ath TX task, rather than directly
calling ath_start().
Now yes, this means that I've taken a step backwards in terms of
concurrency - TX -and- RX now occur in the same single-task taskqueue.
But there's nothing stopping me from separating out the TX / TX completion
code into a separate taskqueue which runs in parallel with the RX path,
if that ends up being appropriate for some platforms.
This fixes the CCMP/seqno concurrency issues that creep up when you
transmit large amounts of uni-directional UDP traffic (>200MBit) on a
FreeBSD STA -> AP, as now there's only one TX context no matter what's
going on (TX completion->retry/software queue,
userland->net80211->ath_start(), TX completion -> ath_start());
but it won't fix any concurrency issues between raw transmitted frames
and non-raw transmitted frames (eg EAPOL frames on TID 16 and any other
TID 16 multicast traffic that gets put on the CABQ.) That is going to
require a bunch more re-architecture before it's feasible to fix.
In any case, this is a big step towards making the majority of the TX
path locking irrelevant, as now almost all TX activity occurs in the
taskqueue.
Phew.
the ATH_TXQ_* macros.
* Introduce the new macros;
* rename the TID queue and TID filtered frame queue so the compiler
tells me I'm using the wrong macro.
These should correspond 1:1 to the existing code.
net80211 node power save state.
* Add an ATH_NODE_UNLOCK_ASSERT() check
* Add a new node field - an_is_powersave
* Pause/unpause the queue based on the node state
* Attempt to handle net80211 concurrency issues so the queue
doesn't get paused/unpaused more than once at a time from
the net80211 power save code.
Whilst here (and breaking my usual rule), set CLRDMASK when a queue
is unpaused, regardless of whether the queue has some pending traffic.
This means the first frame from that TID (now or later) will hvae
CLRDMASK set.
Also whilst here, bump the swretrymax counters whenever the
filtered frames code expires a frame. Again, breaking my rule, but
this is just a statistics thing rather than a functional change.
This doesn't fix ps-poll (but it doesn't break it too much worse
than it is at the present) or correcting the TID updates.
That's next on the list.
Tested:
* AR9220 AP (Atheros AP96 reference design)
* Macbook Pro and LG Optimus 1 Android phone, both setting
and clearing power save state (but not using PS-POLL.)
This should eventually be unified with ATH_DEBUG() so I can get both
from one macro; that may take some time.
Add some new probes for TX and TX completion.
These are intended for software TX filtering support, where the NIC
decides there has been too many successive failues to a destination
and will filter it.
Although the filtering is done per-destination (via the keycache),
the state and queue is kept per-TID for now. It simplifies the overall
architecture design and locking.
Whilst here, add ATH_TID_UNLOCK_ASSERT().
necessary to "do" EDMA.
It was just using the TX completion status for logging information about
the descriptor completion. Since with EDMA we don't know this without
checking the TX completion FIFO, we can't provide this information.
So don't.
Now that I understand what's going on with this, I've realised that
it's going to be quite difficult to implement a processq method in
the EDMA case. Because there's a separate TX status FIFO, I can't
just run processq() on each EDMA TXQ to see what's finished.
i have to actually run the TX status queue and handle individual
TXQs.
So:
* unmethodize ath_tx_processq();
* leave ath_tx_draintxq() as a method, as it only uses the completion status
for debugging rather than actively completing the frames (ie, all frames
here are failed);
* Methodize ath_draintxq().
The EDMA ath_draintxq() will have to take care of running the TX
completion FIFO before (potentially) freeing frames in the queue.
The only two places where ath_tx_draintxq() (on a single TXQ) are used:
* ath_draintxq(); and
* the CABQ handling in the beacon setup code - it drains the CABQ before
populating the CABQ with frames for a new beacon (when doing multi-VAP
operation.)
So it's quite possible that once I methodize the CABQ and beacon handling,
I can just drop ath_tx_draintxq() in its entirety.
Finally, it's also quite possible that I can remove ath_tx_draintxq()
in the future and just "teach" it to not check the status when doing
EDMA.
* Add ATH_TXQ_FIRST() for easy tasting of what's on the list;
* Add an "axq_fifo_depth" for easy tracking of how deep the current
FIFO is;
* Flesh out the handoff (mcast, hw) functions;
* Begin fleshing out a TX ISR proc, which tastes the TX status FIFO.
The legacy hardware stuffs the TX completion at the end of the final frame
descriptor (or final sub-frame when doing aggregate.) So it's feasible
to do a per-TXQ drain and process, as the needed info is right there.
For EDMA hardware, there's a separate TX completion FIFO. So the TX
process routine needs to read the single FIFO and then process the
frames in each hardware queue.
This makes it difficult to do a per-queue process, as you'll end up with
frames in the TX completion FIFO for a different TXQ to the one you've
passed to ath_tx_draintxq() or ath_tx_processq().
Testing:
I've tested the TX queue and TX completion code in hostap mode on an
AR9380. Beacon frames successfully transmit and the completion routine
is called. Occasional data frames end up in TXQ 1 and are also
successfully completed.
However, this requires some changes to the beacon code path as:
* The AR9380 beacon configuration API is now in TU/8, rather than
TU;
* The AR9380 TX API requires the rate control is setup using a call
to setup11nratescenario, rather than having the try0 series setup
(rate/tries for the first series); so the beacon won't go out.
I'll follow this up with commits to the beacon code.
array, similar to what filltxdesc() uses.
This removes the last reference to ds_data in the TX path outside of
debugging statements. These need to be adjusted/fixed.
Tested:
* AR9280 STA/AP with iperf TCP traffic
The existing API only exposes 'seglen' (the current buffer (segment) length)
with the data buffer pointer set in 'ds_data'. This is fine for the legacy
DMA engine but it won't work for the EDMA engines.
The EDMA engine has a significantly different TX descriptor layout.
* The legacy DMA engine had a ds_data pointer at the same offset in the
descriptor for both TX and RX buffers;
* The EDMA engine has no ds_data for RX - the data is DMAed after the
descriptor;
* The EDMA engine has support for 4 TX buffer/segment pairs in the TX
DMA descriptor;
* The EDMA TX completion is in a different FIFO, and the driver will
'link' the status completion entry to a QCU by a "QCU ID".
I don't know why it's just not filled in by the hardware, alas.
So given that, here are the changes:
* Instead of directly fondling 'ds_data' in ath_desc, change the
ath_hal_filltxdesc() to take an array of buffer pointers as well
as segment len pointers;
* The EDMA TX completion status wants a descriptor and queue id.
This (for now) uses bf_state.bfs_txq and will extract the hardware QCU
ID from that.
* .. and this is ugly and wasteful; it should change to just store
the QCU in the bf_state and save 3/7 bytes in the process.
Now, the weird crap:
* The aggregate TX path was using bf_state->bfs_txq for the TXQ, rather than
taking a function argument. I've tidied that up.
* The multicast queue frames get put on a software TXQ and then that is
appended to the hardware CABQ when appropriate. So for now, make sure
that bf_state->bfs_txq points at the CABQ when adding frames to the
multicast queue.
* .. but the multicast queue TX path for now doesn't use the software
queue and instead
(a) directly sets up the descriptor contents at that point;
(b) the frames on the vap->avp_mcastq are then just appended wholesale
to the CABQ.
So for now, I don't have to worry about making the multicast path
work with aggregation or the per-TID software queue. Phew.
What's left to do:
* I need to modify the 11n ath_hal_chaintxdesc() API to do the same.
I'll do that in a subsequent commit.
* Remove bf_state.bfs_txq entirely and store the QCU as appropriate.
* .. then do the runtime "is this going on the right HWQ?" checks using
that, rather than comparing pointer values.
Tested on:
* AR9280 STA/AP
* AR5416 STA/AP
enabled.
The legacy (pre-802.11n) hardware doesn't support this - although
the AR5212 era hardware supports MRR, it doesn't have all the bits
needed to support MRR + RTS/CTS. The AR5416 and later support
a packet duration and RTS/CTS flags per rate scenario, so we should
support it.
Tested:
* AR9280, STA
PR: kern/170302
* Introduce TX DMA setup/teardown methods, mirroring what's done in
the RX path.
Although the TX DMA descriptor is setup via ath_desc_alloc() /
ath_desc_free(), there TX status descriptor ring will be allocated
in this path.
* Remove some of the TX EDMA capability probing from the RX path and
push it into the new TX EDMA path.
sized TX descriptor.
This is required for the AR93xx EDMA support which requires 128 byte
TX descriptors (which is significantly larger than the earlier
hardware.)
The DMA FIFO chips (AR93xx and later) differ slightly to th elegacy
chips:
* The RX DMA descriptors don't have a ds_link field;
* The TX DMA descriptors have a ds_link field however at a different
offset.
This is a reimplementation based on what the reference driver and ath9k
does.
A subsequent commit will enable it in the TX and beacon paths.
Obtained from: Linux ath9k, Qualcomm Atheros
with fresh descriptors, before handling the frames.
Wrap it all in the RX locks.
Since the FIFO is very shallow (16 for HP, 128 for LP) it needs to be
drained and replenished very quickly. Ideally, I'll eventually move this
RX FIFO drain/fill into the interrupt handler, only deferring the actual
frame completion.
The AR93xx and later chips support two RX FIFO queues - a high and low
priority queue.
For legacy chips, just assume the queues are high priority.
This is inspired by the reference driver but is a reimplementation of
the API and code.
The RX EDMA support requires a modified approach to the RX descriptor
handling.
Specifically:
* There's now two RX queues - high and low priority;
* The RX queues are implemented as FIFOs; they're now an array of pointers
to buffers;
* .. and the RX buffer and descriptor are in the same "buffer", rather than
being separate.
So to that end, this commit abstracts out most of the RX related functions
from the bulk of the driver. Notably, the RX DMA/buffer allocation isn't
updated, primarily because I haven't yet fleshed out what it should look
like.
Whilst I'm here, create a set of matching but mostly unimplemented EDMA
stubs.
Tested:
* AR9280, station mode
TODO:
* Thorough AP and other mode testing for non-EDMA chips;
* Figure out how to allocate RX buffers suitable for RX EDMA, including
correctly setting the mbuf length to compensate for the RX descriptor
and completion status area.
as an EDMA check function.
For the AR9003 and later NICs, different TX/RX DMA and descriptor handling
code will be conditional on the EDMA check.
Obtained from: Qualcomm Atheros
* Resize some types. In particular, bfs_seqno can be uint16_t for now.
Previous work would assign the unassigned seqno a value of -1, which
I obviously can't do here.
* Remove bfs_pktdur. It was in the original code but nothing so far uses
it.
This gets ath_buf down (on my i386 system) to 292 bytes from 300 bytes.
I'd rather it be much, much smaller.
ath_start() is called.
This (defaults to 10 frames) gives for a little headway in the TX ath_buf
allocation, so buffer cloning is still possible.
This requires a lot omre experimenting and tuning.
It also doesn't stop a node/TID from consuming all of the available
ath_buf's, especially when the node is going through high packet loss
or only talking at a low TX rate. It also doesn't stop a paused TID
from taking all of the ath_bufs. I'll look at fixing that up in subsequent
commits.
PR: kern/168170
traffic.
* Create sc_mgmt_txbuf and sc_mgmt_txdesc, initialise/free them appropriately.
* Create an enum to represent buffer types in the API.
* Extend ath_getbuf() and _ath_getbuf_locked() to take the above enum.
* Right now anything sent via ic_raw_xmit() allocates via ATH_BUFTYPE_MGMT.
This may not be very useful.
* Add ATH_BUF_MGMT flag (ath_buf.bf_flags) which indicates the current buffer
is a mgmt buffer and should go back onto the mgmt free list.
* Extend 'txagg' to include debugging output for both normal and mgmt txbufs.
* When checking/clearing ATH_BUF_BUSY, do it on both TX pools.
Tested:
* STA mode, with heavy UDP injection via iperf. This filled the TX queue
however BARs were still going out successfully.
TODO:
* Initialise the mgmt buffers with ATH_BUF_MGMT and then ensure the right
type is being allocated and freed on the appropriate list. That'd save
a write operation (to bf->bf_flags) on each buffer alloc/free.
* Test on AP mode, ensure that BAR TX and probe responses go out nicely
when the main TX queue is filled (eg with paused traffic to a TID,
awaiting a BAR to complete.)
PR: kern/168170
This showed up when doing heavy UDP throughput on SMP machines.
The problem with this is because the 802.11 sequence number is being
allocated separately to the CCMP PN replay number (which is assigned
during ieee80211_crypto_encap()).
Under significant throughput (200+ MBps) the TX path would be stressed
enough that frame TX/retry would force sequence number and PN allocation
to be out of order. So once the frames were reordered via 802.11 seqnos,
the CCMP PN would be far out of order, causing most frames to be discarded
by the receiver.
I've fixed this in some local work by being forced to:
(a) deal with the issues that lead to the parallel TX causing out of
order sequence numbers in the first place;
(b) fix all the packet queuing issues which lead to strange (but mostly
valid) TX.
I'll begin fixing these in a subsequent commit or five.
PR: kern/166190
it turns out that it negatively affects performance. I'm stil investigating
exactly why deferring the IO causes such negative TCP performance but
doesn't affect UDP preformance.
Leave the ath_tx_kick() change in there however; it's going to be useful
to have that there for if_transmit() work.
PR: kern/168649
implementing parallel TX and TX/RX completion can be done without
simply abusing long-held locks.
Right now, multiple concurrent ath_start() entries can result in
frames being dequeued out of order. Well, they're dequeued in order
fine, but if there's any preemption or race between CPUs between:
* removing the frame from the ifnet, and
* calling and runningath_tx_start(), until the frame is placed on a
software or hardware TXQ
Then although dequeueing the frame is in-order, queueing it to the hardware
may be out of order.
This is solved in a lot of other drivers by just holding a TX lock over
a rather long period of time. This lets them continue to direct dispatch
without races between dequeue and hardware queue.
Note to observers: if_transmit() doesn't necessarily solve this.
It removes the ifnet from the main path, but the same issue exists if
there's some intermediary queue (eg a bufring, which as an aside also
may pull in ifnet when you're using ALTQ.)
So, until I can sit down and code up a much better way of doing parallel
TX, I'm going to leave the TX path using a deferred taskqueue task.
What I will likely head towards is doing a direct dispatch to hardware
or software via if_transmit(), but it'll require some driver changes to
allow queues to be made without using the really large ath_buf / ath_desc
entries.
TODO:
* Look at how feasible it'll be to just do direct dispatch to
ath_tx_start() from if_transmit(), avoiding doing _any_ intermediary
serialisation into a global queue. This may break ALTQ for example,
so I have to be delicate.
* It's quite likely that I should break up ath_tx_start() so it
deposits frames onto the software queues first, and then only fill
in the 802.11 fields when it's being queued to the hardware.
That will make the if_transmit() -> software queue path very
quick and lightweight.
* This has some very bad behaviour when using ACPI and Cx states.
I'll do some subsequent analysis using KTR and schedgraph and file
a follow-up PR or two.
PR: kern/168649
