Allow TLS records to be decrypted in the kernel after being received
by a NIC. At a high level this is somewhat similar to software KTLS
for the transmit path except in reverse. Protocols enqueue mbufs
containing encrypted TLS records (or portions of records) into the
tail of a socket buffer and the KTLS layer decrypts those records
before returning them to userland applications. However, there is an
important difference:
- In the transmit case, the socket buffer is always a single "record"
holding a chain of mbufs. Not-yet-encrypted mbufs are marked not
ready (M_NOTREADY) and released to protocols for transmit by marking
mbufs ready once their data is encrypted.
- In the receive case, incoming (encrypted) data appended to the
socket buffer is still a single stream of data from the protocol,
but decrypted TLS records are stored as separate records in the
socket buffer and read individually via recvmsg().
Initially I tried to make this work by marking incoming mbufs as
M_NOTREADY, but there didn't seemed to be a non-gross way to deal with
picking a portion of the mbuf chain and turning it into a new record
in the socket buffer after decrypting the TLS record it contained
(along with prepending a control message). Also, such mbufs would
also need to be "pinned" in some way while they are being decrypted
such that a concurrent sbcut() wouldn't free them out from under the
thread performing decryption.
As such, I settled on the following solution:
- Socket buffers now contain an additional chain of mbufs (sb_mtls,
sb_mtlstail, and sb_tlscc) containing encrypted mbufs appended by
the protocol layer. These mbufs are still marked M_NOTREADY, but
soreceive*() generally don't know about them (except that they will
block waiting for data to be decrypted for a blocking read).
- Each time a new mbuf is appended to this TLS mbuf chain, the socket
buffer peeks at the TLS record header at the head of the chain to
determine the encrypted record's length. If enough data is queued
for the TLS record, the socket is placed on a per-CPU TLS workqueue
(reusing the existing KTLS workqueues and worker threads).
- The worker thread loops over the TLS mbuf chain decrypting records
until it runs out of data. Each record is detached from the TLS
mbuf chain while it is being decrypted to keep the mbufs "pinned".
However, a new sb_dtlscc field tracks the character count of the
detached record and sbcut()/sbdrop() is updated to account for the
detached record. After the record is decrypted, the worker thread
first checks to see if sbcut() dropped the record. If so, it is
freed (can happen when a socket is closed with pending data).
Otherwise, the header and trailer are stripped from the original
mbufs, a control message is created holding the decrypted TLS
header, and the decrypted TLS record is appended to the "normal"
socket buffer chain.
(Side note: the SBCHECK() infrastucture was very useful as I was
able to add assertions there about the TLS chain that caught several
bugs during development.)
Tested by: rmacklem (various versions)
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24628
This avoids injecting errors into the test system's mirrors.
gnop seems like a good solution here but it injects errors at the wrong
place vs where these tests expect and does not support a 'max global count'
like the failpoints do with 'n*' syntax.
Reviewed by: cem, vangyzen
Sponsored by: Dell EMC Isilon
This function manages the loop around crypto_dispatch and coordination
with ktls_ocf_callback.
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D25757
The size of the containing structure was passed instead of the size of
the array. This happened to be harmless as the extra word copied is
one we copy in the next line anyway.
Reported by: CHERI (bounds check violation)
Reviewed by: brooks, imp
Obtained from: CheriBSD
MFC after: 1 week
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D25791
an ungraceful operation.
This fixes a use-after-free bug found and reported by Taylor
Brandstetter of Google by testing the userland stack.
MFC after: 1 week
It is apparently broken when assembled by contemporary GNU as as well as
Clang IAS (which is used in the default configuration).
PR: 248221
Reported by: pizzamig
Sponsored by: The FreeBSD Foundation
In such a case the second argument to lock_delay_arg_init was NULL which was
immediately causing a null pointer deref.
Since the sructure is only used for spin count, provide a dedicate routine
initializing it.
Reported by: andrew
AVL trees, red-black trees, and others. Weak AVL (wavl) trees are a
recently discovered member of that class. This change replaces
red-black rebalancing with weak AVL rebalancing in the RB tree macros.
Wavl trees sit between AVL and red-black trees in terms of how
strictly balance is enforced. They have the stricter balance of AVL
trees as the tree is built - a wavl tree is an AVL tree until the
first deletion. Once removals start, wavl trees are lazier about
rebalancing than AVL trees, so that removals can be fast, but the
balance of the tree can decay to that of a red-black tree. Subsequent
insertions can push balance back toward the stricter AVL conditions.
Removing a node from a wavl tree never requires more than two
rotations, which is better than either red-black or AVL
trees. Inserting a node into a wavl tree never requires more than two
rotations, which matches red-black and AVL trees. The only
disadvantage of wavl trees to red-black trees is that more insertions
are likely to adjust the tree a bit. That's the cost of keeping the
tree more balanced.
Testing has shown that for the cases where red-black trees do worst,
wavl trees better balance leads to faster lookups, so that if lookups
outnumber insertions by a nontrivial amount, lookup time saved exceeds
the extra cost of balancing.
Reviewed by: alc, gbe, markj
Tested by: pho
Discussed with: emaste
Differential Revision: https://reviews.freebsd.org/D25480
In original implementation, zpool history will read the whole history
before printing anything, causing memory usage goes unbounded. We fix
this by breaking it into read-print iterations.
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Chunwei Chen <david.chen@nutanix.com>
Closes#9516
Note, this change changes the libzfs.so ABI by modifying the prototype
of zpool_get_history(). Since libzfs is effectively private to the base
system it is anticipated that this will not be a problem.
PR: 247557
Obtained from: OpenZFS
Reported and tested by: Sam Vaughan <samjvaughan@gmail.com>
Discussed with: freqlabs
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D25745openzfs/zfs@7125a109dc
The counter keeps being updated all the time and variables read afterwards
share the cacheline. Note this still fundamentally does not scale and needs
to be replaced, in the meantime gets a bandaid.
brk1_processes -t 52 ops/s:
before: 8598298
after: 9098080
Way back in r28911 (August 1997, CVS rev 1.22) we imported a NetBSD
information leak fix via OpenBSD. Unfortunatly we failed to track the
followup commit that fixed the type of the error code. Apply the change
from int to long now.
Reviewed by: emaste
Found by: CHERI
Obtained from: CheriBSD
MFC after: 3 days
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D25779
This patch modifies writing to mirrored pNFS DSs slightly so that there is
only one m_copym() call for a mirrored pair instead of two of them.
This call replaces the custom nfsm_copym() call, which is no longer needed
and deleted by this patch. The patch does introduce a new nfsm_split()
function that only calls m_split() for the non-ext_pgs case.
The semantics of nfsm_uiombuflist() is changed to include code that nul
pads the generated mbuf list. This was done by nfsm_copym() prior to this patch.
The main reason for this change is that it allows the data to be a list
of ext_pgs mbufs, since the m_copym() is for the entire mbuf list.
This support will be added in a future commit.
This patch only affects writing to mirrored flexible file layout pNFS servers.
This happens when there's only one component (e.g. "/foo"). This
(mostly-harmless) bug has been present since June 1990 when it was
commited to mountd.c SCCS version 5.9.
Note: the bug is on the second changed line, the first line is changed
for visual consistency.
Reviewed by: cem, emaste, mckusick, rmacklem
Found with: CHERI
Obtained from: CheriBSD
MFC after: 1 week
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D25759
The only part of nmi_handle_intr() depending on ISA is isa_nmi(), which is
already wrapped. Entering debugger on NMI does not really depend on ISA.
MFC after: 2 weeks
This is a generic function start a scan request for the given
cam_sim.
Other driver can now just use this function to request a new rescan.
Submitted by: kibab
Right now, the bootstrap will gloss over things like pkg bootstrap -x or
pkg bootstrap -f pkg. Make it more clear that this is incorrect, and hint
at the correct formatting.
Reported by: jhb (IIRC via IRC)
Approved by: bapt, jhb, manu
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D24750
Handle the fact that parts of usb(4) can be compiled into the boot
loader, where M_WAITOK does not guarantee a successful allocation.
PR: 240545
Submitted by: Andrew Reiter <arr@watson.org> (original version)
Reviewed by: hselasky
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D25706
Add tests to cover "add", "change" and "delete" functionality of /sbin/route
for ipv4 and ipv6. These tests for the existing route tool are the first step
towards creating libroute.
Submitted by: Ahsan Barkati
Sponsored by: Google, Inc. (GSoC 2020)
Reviewed by: kp, thj
Approved by: bz (mentor)
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D25220
It is very conservative. Only spinning when LK_ADAPTIVE is passed, only on
exclusive lock and never when any waiters are present. buffer cache is remains
not spinning.
This reduces total sleep times during buildworld etc., but it does not shorten
total real time (culprits are contention in the vm subsystem along with slock +
upgrade which is not covered).
For microbenchmarks: open3_processes -t 52 (open/close of the same file for
writing) ops/s:
before: 258845
after: 801638
Reviewed by: kib
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D25753
lbh is included for consistency with other functions and in case
future work needs to use it, but it is currently unused. Mark it,
and a post-OpenZFS-import world will be able to raise WARNS of
libbe to the default (pending some minor changes to openzfs libzfs).
MFC after: 3 days
During device attachment, all interrupt sources will bind to the BSP,
as it is the only processor online. This means interrupts must be
redistributed ("shuffled") later, during SI_SUB_SMP.
For the EARLY_AP_STARTUP case, this is no longer true. SI_SUB_SMP will
execute much earlier, meaning APs will be online and available before
devices begin attachment, and there will therefore be nothing to
shuffle.
All PIC-conforming interrupt controllers will handle this early
distribution properly, except for RISC-V's PLIC. Make the necessary
tweak to the PLIC driver.
While here, convert irq_assign_cpu from a boolean_t to a bool.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D25693
Remove all variations of rtrequest <rtrequest1_fib, rtrequest_fib,
in6_rtrequest, rtrequest_fib> and their uses and switch to
to rib_action(). This is part of the new routing KPI.
Submitted by: Neel Chauhan <neel AT neelc DOT org>
Differential Revision: https://reviews.freebsd.org/D25546
10.0.1 final (aka llvmorg-10.0.1-0-gef32c611aa2).
There were no changes since rc2, except in the upstream regression
tests, which we do not ship.
Relnotes: yes
MFC after: immediately (no material changes except tag)
