- Make use of cursors to avoid data copies for AES-CCM and AES-GCM.
Pass pointers into the request's input and/or output buffers
directly to the Update, encrypt, and decrypt hooks rather than
always copying all data into a temporary block buffer on the stack.
- Move handling for partial final blocks out of the main loop.
This removes branches from the main loop and permits using
encrypt/decrypt_last which avoids a memset to clear the rest of the
block on the stack.
- Shrink the on-stack buffers to assume AES block sizes and CCM/GCM
tag lengths.
- For AAD data, pass larger chunks to axf->Update. CCM can take each
AAD segment in a single call. GMAC can take multiple blocks at a
time.
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D25058
The amount to copy for the first block is the minimum of the size of
the AAD region or the remaining space in the first block.
Reported by: cryptocheck -z
MFC after: 2 weeks
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D25140
- Clear the current thread's TLS pointer on exec. Previously the TLS
pointer (and register) remain unchanged.
- Explicitly clear the TLS pointer when new threads are created.
- Make md_tls_tcb_offset per-process instead of per-thread.
The layout of the TLS and TCB are identical for all threads in a
process, it is only the TLS pointer values themselves that vary by
thread. This also makes setting md_tls_tcb_offset in
cpu_set_user_tls() redundant with the setting in exec_setregs(), so
only set it in exec_setregs().
Submitted by: Alfredo Mazzinghi (1)
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24957
The Intel Instruction Set Reference says this about the XSAVE instruction:
Use of a destination operand not aligned to 64-byte boundary
(in either 64-bit or 32-bit modes) results in a general-protection
(#GP) exception.
This alignment happens naturally when all malloc buckets are powers
of two. However, this change is necessary on some systems when
certain non-power-of-two (and non-multiple of 64) malloc buckets
are defined.
Reviewed by: cem; kib; earlier version by jhb
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D25098
In particular, uma_zcreate creates sysctl oids, which locks an sx lock,
which uses IPIs under contention. IPIs tend not to work very well
when interrupts are disabled. Who knew, right?
Reviewed by: cem kib
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D25098
always want to do a window update, even when no data can be sent. Now in
cases where you are not pacing thats probably ok, you just send an extra
window update or two. However with bbr (and rack if its paced) every time
the pacer goes off its going to send a "window update".
Also in testing bbr I have found that if we are not responding to
data right away we end up staying in startup but incorrectly holding
a pacing gain of 192 (a loss). This is because the idle window code
does not restict itself to only work with PROBE_BW. In all other
states you dont want it doing a PROBE_BW state change.
Sponsored by: Netflix Inc.
Differential Revision: https://reviews.freebsd.org/D25247
Given that 64c/128t CPUs are currently available, and that many
devices (nvme, many NICs) desire to map 1 MSI-X vector per core,
or even 1 per-thread, it is becoming far easier to see MSI-X interrupt
setup fail due to msi vector exhaustion, and devices fail to attach at
boot on large system.
This bump costs 12KB on amd64 (and 6KB on i386), which seems
worth the trade off for a better out of the box experience on
high end hardware.
Reviewed by: jhb
MFC after: 21 days
Sponsored by: Netflix
RB_LEFT or RB_RIGHT, so they aren't stripping off the color bit
encoded there. Strip off that bit for linuxkpi.
Reported by: dch
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D25245
the debug messages. While here, clean up some variable naming.
Reviewed by: bcr (manpages), emaste
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D25230
When doing A-MSDU offload handling the driver is required to mark
A-MSDUs from the same MPDU with the same sequence number.
It then tags them as AMSDU (if it's a decap'ed A-MSDU) and AMSDU_MORE
(saying there's more AMSDUs decapped in the same MSDU.)
This allows encryption and sequence number offload to work right.
In the A-MSDU path the sequence number check looks at the A-MSDU flags
in the frame to see whether it's part of the same seqno and will pass them
(ie, not increment rx_seq until the last A-MSDU is seen from the driver,
or a new seqno shows up.0
However, I did this work in the A-MSDU path but not the A-MSDU in A-MPDU path.
For the non A-MDSU offload case the A-MPDU receive reordering will do its
thing and then pass up the MPDU up for decap - which then will see it's
an A-MSDU and decap each sub-frame. But this isn't done for offloaded
A-MSDU frames.
This requires two parts:
* Don't bump the RX sequence number, same as above; and
* If frames go into the reordering buffer, they need to be added into the slot
as a set of frames rather than a single frame, so once a new seqno shows up
this slot can be marked as "full" and we can move on.
This patch does the first. The latter requires that I find and commit
work to change rxa_m from an mbuf to an mbufq and the nhandle A-MSDU
there. But, the first is enough to allow the normal case (ie, no or not
a lot of A-MPDU RX reordering) to work.
This allows the athp driver (QCA9880) throughput to go from VERY low
(like 5mbit TCP, 1/3-1/4 expected UDP throughput) to ~ 250mbit TCP
and > 300mbit UDP on a VHT/40 channel. TCP sucks because, well, it
shows up as MASSIVE packet loss when all but one frame in a decap'ed
A-MSDU stream is dropped. Le whoops.
Now, where'd I put that laptop with the patch for rxa_m mbufq that
I wrote like in 2017...
Tested:
* AR9380, STA/AP mode (a big no-op, no A-MSDU hardware decap);
* if_run (RT3593), STA DWDS mode (A-MPDU / A-MSDU receive, but again
no A-MSDU hardware decap);
* QCA9880, STA/AP mode (which is doing hardware A-MPDU/A-MSDU decap,
but no A-MPDU reordering in the firmware.)
r286700 added the "lacp_fast_timeout" option to `ifconfig', but we forgot to
include the new option in the string used to decode the option bits. Add
"LACP_FAST_TIMO" to LAGG_OPT_BITS.
Also, s/LAGG_OPT_LACP_TIMEOUT/LAGG_OPT_LACP_FAST_TIMO/g , to be clearer that
the flag indicates "Fast Timeout" mode.
Reported by: Greg Foster <gfoster at panasas dot com>
Reviewed by: jpaetzel
MFC after: 1 week
Sponsored by: Panasas
Differential Revision: https://reviews.freebsd.org/D25239
This function returns NULL if the ring identified by
queue id and direction is in netmap mode. Otherwise
return the corresponding kring.
Use this function to replace vtnet_netmap_queue_on().
MFC after: 1 week
In particular, invalidation of the preloaded modules text to allow
execution from it was broken after D25188/r362031.
Reviewed by: markj
Reported by: delphij, dhw
Sponsored by: The FreeBSD Foundation
MFC after: 13 days
This partially reverts r361053 since there have been reports
by users that this breaks some functionality for em(4)
devices; it seems at first glance that some sort of interface
restart is required for those cards.
This isn't a proper fix; this unbreaks those users until a proper
fix is found for their issues.
PR: 240818
Reported by: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl>
MFC after: 3 days
With this change all ZVOL updates are initiated from the SPA sync
context instead of a mix of the sync and open contexts. The updates are
queued to be applied by a dedicated thread in the original order. This
should ensure that ZVOLs always accurately reflect the corresponding
datasets. ZFS ioctl operations wait on the mentioned thread to complete
its work. Thus, the illusion of the synchronous ZVOL update is
preserved. At the same time, the SPA sync thread never blocks on ZVOL
related operations avoiding problems like reported in bug 203864.
This change is based on earlier work in the same direction: D7179 and
D14669 by Anthoine Bourgeois. D7179 tried to perform ZVOL operations
in the open context and that opened races between them. D14669 uses a
design very similar to this change but with different implementation
details.
This change also heavily borrows from similar code in ZoL, but there are
many differences too. See:
- a0bd735adb
- https://github.com/zfsonlinux/zfs/issues/3681
- https://github.com/zfsonlinux/zfs/issues/2217
PR: 203864
MFC after: 5 weeks
Sponsored by: CyberSecure
Differential Revision: https://reviews.freebsd.org/D23478
Allow the TCP header to reside in the mbuf following the IP header.
Else such packets will get dropped.
Backtrace:
mlx5e_sq_xmit()
mlx5e_xmit()
ether_output_frame()
ether_output()
ip_output_send()
ip_output()
rip_output()
sosend_generic()
sosend()
kern_sendit()
sendit()
sys_sendto()
amd64_syscall()
fast_syscall_common()
MFC after: 1 week
Sponsored by: Mellanox Technologies
Typically the TCP/IP headers fit within the first mbuf and should not
trigger any of the error cases. Use unlikely() for these cases.
No functional change.
MFC after: 1 week
Sponsored by: Mellanox Technologies
When parsing the TCP/IP header in the fast path, make it clear by using
the const keyword, no fields are to be modified inside the transmitted
packet.
No functional change.
MFC after: 1 week
Sponsored by: Mellanox Technologies
I2C_SET was quite inflexible, it used too long delays as well as some
unnecessary delays. The new building blocks are iicbb_clockin and
iicbb_clockout. The former sets SDA and starts the high period of SCL,
the latter executes the low period of SCL. What happens during the high
phase depends on the operation. For writes we just hold both lines, for
reads we poll SDA. S, Sr and P change SDA in the middle of the high
period.
Also, the calculation of udelay has been updated, so that the resulting
period more closely corresponds the requested bus frequency. There is a
new knob, io_delay, that allows to further adjust udelay based on the
estimated latency of pin toggling operations.
Finally, I slightly changed debug tracing and added error indicators to
it. The debug prints are compiled in but disabled by default. This can
be of use if there is any fallout from this change.
Some ideas for further improvements:
- add a function for sub-microsecond delays (e.g., in units of 1/10th of
a microsecond) and use it for more precise timing of short delays;
- account for the actual time spent in the pin I/O.
Some sample debug output with the new code follows.
Reading temperature and humidity from HTU21 in the bus hold mode:
<<w80+ we3+ <w81+ .....r6d+ rac+ r94- >>
<<w80+ we5+ <w81+ .............r47+ re2+ r84- >>
where '<<' is S, '<' is Sr, '>>' is P, '.' is one millisecond of clock
stretching by the slave.
Reading temperature and humidity in the no-hold mode:
<<w80+ wf3+ >>
<<w81- >>
<<w81+ r6d+ r54+ raf- >>
<<w80+ wf5+ >>
<<w81- >>
<<w81+ r48+ r4e+ r9c- >>
where '+' is Ack and '-' is NoAck.
We see that first read attempts are not acknowledged.
MFC after: 4 weeks
Differential Revision: https://reviews.freebsd.org/D22206
Like every other firmware image in the tree, the makefile will need to
be updated to point to the newest import.
Reviewed by: erj, imp (previous version)
Differential Revision: https://reviews.freebsd.org/D25222
- Use the same definition of free memory as Linux.
- Rename the totalbig and freebig fields to match the corresponding
names on Linux.
Discussed with: alc
MFC after: 1 week
The firmware module arbitrarily limits us to at most 50 images. It is
possible to hit this limit on platforms that preload many firmware
images, or link all of the firmware images for a set of devices into the
kernel.
Convert the table into a linked list, removing the limit.
Reported by: Steve Wheeler
Reviewed by: rpokala
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC (Netgate)
Differential Revision: https://reviews.freebsd.org/D25161
