I'm still not sure whether this is the full solution, but here goes.
I have a two node DWDS setup - a main AP with the ethernet bridge uplink
and a satellite AP in the back of the house. They're both AR9344+AR9580
dual band 11n APs.
The problem was that multicast frames was not going from the DWDS AP to
the DWDS STA. Unicast frames are fine, and multicast frames from the
DWDS STA to AP are fine.
Now, multicast and unicast frames from the STA -> AP are just transmitted
using the unicast key. That's fine. However, the AP -> STA multicast
frames by default are transmitted using the current default / multicast
key, the shared one between all STAs in a BSS. Now, the DWDS implementation
ignores non WDS frames - it only allows about 4 address frames outside
of management / EAPOL frames! - so the STA side ignores the normal multicast
frames.
Instead, the AP side uses ieee80211_dwds_mcast() to send multicast frames
to each WDS VAP that was created as part of the "dynamic" part of DWDS.
This should be queuing them individually to each node instead of using
the normal multicast send path; and this is how they should get turned into
4-addr WDS frames.
HOWEVER, ieee80211_encap() was trying to use the default TX key to queue
them rather than the unicast key that's already setup. Since this synthetic
node doesn't have the default TX key setup, transmission fails. Things
would be fine in WEP and in open mode because in both cases you would
have static keys (or no keys) setup. It just fails in WPA mode.
This resolves the issue. AP DWDS multicast is now sent using the unicast
key just like in STA mode and I'm pretty sure the STA mode side will stil
work fine (as it's a STA VAP with a DWDS flag..)
Tested:
* TL-WDR3600/4300 APs
f91b94f22b