Commit Graph

297 Commits

Author SHA1 Message Date
John Baldwin
86be314d09 cryptosoft: Use crypto_cursor_segment().
Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30446
2021-05-25 16:59:19 -07:00
John Baldwin
beb817edfe crypto: Add crypto_cursor_segment() to fetch both base and length.
This function combines crypto_cursor_segbase() and
crypto_cursor_seglen() into a single function.  This is mostly
beneficial in the unmapped mbuf case where back to back calls of these
two functions have to iterate over the sub-components of unmapped
mbufs twice.

Bump __FreeBSD_version for crypto drivers in ports.

Suggested by:	markj
Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30445
2021-05-25 16:59:19 -07:00
John Baldwin
6b313a3a60 Include the trailer in the original dst_iov.
This avoids creating a duplicate copy on the stack just to
append the trailer.

Reviewed by:	gallatin, markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30139
2021-05-25 16:59:19 -07:00
John Baldwin
21e3c1fbe2 Assume OCF is the only KTLS software backend.
This removes support for loadable software backends.  The KTLS OCF
support is now always included in kernels with KERN_TLS and the
ktls_ocf.ko module has been removed.  The software encryption routines
now take an mbuf directly and use the TLS mbuf as the crypto buffer
when possible.

Bump __FreeBSD_version for software backends in ports.

Reviewed by:	gallatin, markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30138
2021-05-25 16:59:19 -07:00
John Baldwin
4a92afae7f ktls_ocf: Fix a few places to not hardcode the GMAC hash length.
This is not a functional change as the Poly1305 hash is the same
length as the GMAC hash length.

Reviewed by:	gallatin, markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30137
2021-05-25 16:59:18 -07:00
John Baldwin
883a0196b6 crypto: Add a new type of crypto buffer for a single mbuf.
This is intended for use in KTLS transmit where each TLS record is
described by a single mbuf that is itself queued in the socket buffer.
Using the existing CRYPTO_BUF_MBUF would result in
bus_dmamap_load_crp() walking additional mbufs in the socket buffer
that are not relevant, but generating a S/G list that potentially
exceeds the limit of the tag (while also wasting CPU cycles).

Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30136
2021-05-25 16:59:18 -07:00
John Baldwin
1c8f4b3c9f Support unmapped mbufs in crypto buffers.
Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30134
2021-05-25 16:59:18 -07:00
Mark Johnston
1a04f0156c cryptodev: Fix some input validation bugs
- When we do not have a separate IV, make sure that the IV length
  specified by the session is not larger than the payload size.
- Disallow AEAD requests without a separate IV.  crp_sanity() asserts
  that CRYPTO_F_IV_SEPARATE is set for AEAD requests, and some (but not
  all) drivers require it.
- Return EINVAL for AEAD requests if an IV is specified but the
  transform does not expect one.

Reported by:	syzbot+c9e8f6ff5cb7fa6a1250@syzkaller.appspotmail.com
Reported by:	syzbot+007341439ae295cee74f@syzkaller.appspotmail.com
Reported by:	syzbot+46e0cc42a428b3b0a40d@syzkaller.appspotmail.com
Reported by:	syzbot+2c4d670173b8bdb947df@syzkaller.appspotmail.com
Reported by:	syzbot+220faa5eeb4d47b23877@syzkaller.appspotmail.com
Reported by:	syzbot+e83434b40f05843722f7@syzkaller.appspotmail.com
Reviewed by:	jhb
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D30154
2021-05-11 17:36:12 -04:00
John Baldwin
76681661be OCF: Remove support for asymmetric cryptographic operations.
There haven't been any non-obscure drivers that supported this
functionality and it has been impossible to test to ensure that it
still works.  The only known consumer of this interface was the engine
in OpenSSL < 1.1.  Modern OpenSSL versions do not include support for
this interface as it was not well-documented.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D29736
2021-04-12 14:28:43 -07:00
John Baldwin
86e352c934 Fix a typo in a comment: frame -> framework.
MFC after:	1 week
Sponsored by:	Chelsio Communications
2021-04-09 16:10:55 -07:00
John Baldwin
3fa034210c ktls: Fix non-inplace TLS 1.3 encryption.
Copy the iovec for the trailer from the proper place.  This is the same
fix for CBC encryption from ff6a7e4ba6.

Reported by:	gallatin
Reviewed by:	gallatin, markj
Fixes:		49f6925ca
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D29177
2021-03-10 11:07:40 -08:00
Mark Johnston
4fc60fa929 opencrypto: Make cryptosoft attach silently
cryptosoft is always present and doesn't print any useful information
when it attaches.

Reviewed by:	jhb
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D29098
2021-03-05 13:11:25 -05:00
John Baldwin
bb6e84c988 poly1305: Don't export generic Poly1305_* symbols from xform_poly1305.c.
There currently isn't a need to provide a public interface to a
software Poly1305 implementation beyond what is already available via
libsodium's APIs and these symbols conflict with symbols shared within
the ossl.ko module between ossl_poly1305.c and ossl_chacha20.c.

Reported by:	se, kp
Fixes:		78991a93eb
Sponsored by:	Netflix
2021-03-05 09:55:11 -08:00
Mark Johnston
ff6a7e4ba6 ktls: Fix CBC encryption when input and output iov sizes are different
Reported by:	gallatin
Tested by:	gallatin
Fixes:		49f6925ca
Differential Revision:	https://reviews.freebsd.org/D29073
2021-03-04 22:45:40 -05:00
Mark Johnston
49f6925ca3 ktls: Cache output buffers for software encryption
Maintain a cache of physically contiguous runs of pages for use as
output buffers when software encryption is configured and in-place
encryption is not possible.  This makes allocation and free cheaper
since in the common case we avoid touching the vm_page structures for
the buffer, and fewer calls into UMA are needed.  gallatin@ reports a
~10% absolute decrease in CPU usage with sendfile/KTLS on a Xeon after
this change.

It is possible that we will not be able to allocate these buffers if
physical memory is fragmented.  To avoid frequently calling into the
physical memory allocator in this scenario, rate-limit allocation
attempts after a failure.  In the failure case we fall back to the old
behaviour of allocating a page at a time.

N.B.: this scheme could be simplified, either by simply using malloc()
and looking up the PAs of the pages backing the buffer, or by falling
back to page by page allocation and creating a mapping in the cache
zone.  This requires some way to save a mapping of an M_EXTPG page array
in the mbuf, though.  m_data is not really appropriate.  The second
approach may be possible by saving the mapping in the plinks union of
the first vm_page structure of the array, but this would force a vm_page
access when freeing an mbuf.

Reviewed by:	gallatin, jhb
Tested by:	gallatin
Sponsored by:	Ampere Computing
Submitted by:	Klara, Inc.
Differential Revision:	https://reviews.freebsd.org/D28556
2021-03-03 17:34:01 -05:00
John Baldwin
a10020cfe2 cryptosoft: Support per-op keys for AES-GCM and AES-CCM.
Reviewed by:	cem
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D28752
2021-02-18 09:53:25 -08:00
John Baldwin
4dd6800e22 Add Chacha20-Poly1305 support in the OCF backend for KTLS.
This supports Chacha20-Poly1305 for both send and receive for TLS 1.2
and for send in TLS 1.3.

Reviewed by:	gallatin
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D27841
2021-02-18 09:30:13 -08:00
John Baldwin
dd2e1352b6 Add an implementation of CHACHA20_POLY1305 to cryptosoft.
This uses the chacha20 IETF and poly1305 implementations from
libsodium.  A seperate auth_hash is created for the auth side whose
Setkey method derives the poly1305 key from the AEAD key and nonce as
described in RFC 8439.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D27837
2021-02-18 09:26:23 -08:00
John Baldwin
fc8fc743d8 Add an OCF algorithm for ChaCha20-Poly1305 AEAD.
Note that this algorithm implements the mode defined in RFC 8439.

Reviewed by:	cem
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D27836
2021-02-18 09:26:00 -08:00
Mark Johnston
db6b56441e ktls: Avoid wakeups and locking for synchronous callbacks
When performing encryption in software, the KTLS crypto callback always
locks the session to deliver a wakeup.  But, if we're handling the
operation synchronously this is wasted effort and can result in
sleepqueue lock contention on large systems.

Use CRYPTO_SESS_SYNC() to determine whether the operation will be
completed asynchronously or not, and select a callback appropriately.
Avoid locking the session to check for completion if the session handles
requests synchronously.

Reviewed by:	jhb
Sponsored by:	Ampere Computing
Submitted by:	Klara, Inc.
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D28195
2021-02-08 09:19:27 -05:00
Mark Johnston
68f6800ce0 opencrypto: Introduce crypto_dispatch_async()
Currently, OpenCrypto consumers can request asynchronous dispatch by
setting a flag in the cryptop.  (Currently only IPSec may do this.)   I
think this is a bit confusing: we (conditionally) set cryptop flags to
request async dispatch, and then crypto_dispatch() immediately examines
those flags to see if the consumer wants async dispatch. The flag names
are also confusing since they don't specify what "async" applies to:
dispatch or completion.

Add a new KPI, crypto_dispatch_async(), rather than encoding the
requested dispatch type in each cryptop. crypto_dispatch_async() falls
back to crypto_dispatch() if the session's driver provides asynchronous
dispatch. Get rid of CRYPTOP_ASYNC() and CRYPTOP_ASYNC_KEEPORDER().

Similarly, add crypto_dispatch_batch() to request processing of a tailq
of cryptops, rather than encoding the scheduling policy using cryptop
flags.  Convert GELI, the only user of this interface (disabled by
default) to use the new interface.

Add CRYPTO_SESS_SYNC(), which can be used by consumers to determine
whether crypto requests will be dispatched synchronously. This is just
a helper macro. Use it instead of looking at cap flags directly.

Fix style in crypto_done(). Also get rid of CRYPTO_RETW_EMPTY() and
just check the relevant queues directly. This could result in some
unnecessary wakeups but I think it's very uncommon to be using more than
one queue per worker in a given workload, so checking all three queues
is a waste of cycles.

Reviewed by:	jhb
Sponsored by:	Ampere Computing
Submitted by:	Klara, Inc.
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D28194
2021-02-08 09:19:19 -05:00
Mark Johnston
1755b2b989 ktls: Use COUNTER_U64_DEFINE_EARLY
This makes it a bit more straightforward to add new counters when
debugging.  No functional change intended.

Reviewed by:	jhb
Sponsored by:	Ampere Computing
Submitted by:	Klara, Inc.
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D28498
2021-02-08 09:18:51 -05:00
Mark Johnston
8adcc757b8 opencrypto: Add comments describing the new crypto_session layout
Requested by:	rpokala
2021-01-19 21:32:33 -05:00
Mark Johnston
98d788c867 opencrypto: Fix assignment of crypto completions to worker threads
Since r336439 we simply take the session pointer value mod the number of
worker threads (ncpu by default).  On small systems this ends up
funneling all completion work through a single thread, which becomes a
bottleneck when processing IPSec traffic using hardware crypto drivers.
(Software drivers such as aesni(4) are unaffected since they invoke
completion handlers synchonously.)

Instead, maintain an incrementing counter with a unique value per
session, and use that to distribute work to completion threads.

Reviewed by:	cem, jhb
MFC after:	2 weeks
Sponsored by:	Rubicon Communications, LLC ("Netgate")
Differential Revision:	https://reviews.freebsd.org/D28159
2021-01-19 20:34:35 -05:00
Mark Johnston
d181624889 opencrypto: Embed the driver softc in the session structure
Store the driver softc below the fields owned by opencrypto.  This is
a bit simpler and saves a pointer dereference when fetching the driver
softc when processing a request.

Get rid of the crypto session UMA zone.  Session allocations are
frequent or performance-critical enough to warrant a dedicated zone.

No functional change intended.

Reviewed by:	cem, jhb
Sponsored by:	Rubicon Communications, LLC ("Netgate")
Differential Revision:	https://reviews.freebsd.org/D28158
2021-01-19 20:34:35 -05:00
John Baldwin
688f8b822c Remove the cloned file descriptors for /dev/crypto.
Crypto file descriptors were added in the original OCF import as a way
to provide per-open data (specifically the list of symmetric
sessions).  However, this gives a bit of a confusing API where one has
to open /dev/crypto and then invoke an ioctl to obtain a second file
descriptor.  This also does not match the API used with /dev/crypto on
other BSDs or with Linux's /dev/crypto driver.

Character devices have gained support for per-open data via cdevpriv
since OCF was imported, so use cdevpriv to simplify the userland API
by permitting ioctls directly on /dev/crypto descriptors.

To provide backwards compatibility, CRIOGET now opens another
/dev/crypto descriptor via kern_openat() rather than dup'ing the
existing file descriptor.  This preserves prior semantics in case
CRIOGET is invoked multiple times on a single file descriptor.

Reviewed by:	markj
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27302
2020-11-25 00:10:54 +00:00
John Baldwin
3acf4d2374 Use void * in place of caddr_t.
Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27065
2020-11-06 18:09:52 +00:00
John Baldwin
c423784dc5 Group session management routines together before first use.
- Rename cse*() to cse_*() to more closely match other local APIs in
  this file.

- Merge the old csecreate() into cryptodev_create_session() and rename
  the new function to cse_create().

Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27070
2020-11-06 18:05:29 +00:00
John Baldwin
f5074add75 Move cryptof_ioctl() below the routines it calls.
Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27069
2020-11-06 00:15:52 +00:00
John Baldwin
b19d4c075f Split logic to create new sessions into a separate function.
This simplifies cryptof_ioctl as it now a wrapper around functions that
contain the bulk of the per-ioctl logic.

Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27068
2020-11-06 00:10:58 +00:00
John Baldwin
c54004c6a9 Move cryptodev_cb earlier before it is used.
This is consistent with cryptodevkey_cb being defined before it is used
and removes a prototype in the middle of the file.

Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27067
2020-11-05 23:42:36 +00:00
John Baldwin
5973f4922d Style fixes for function prototypes and definitions.
Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27066
2020-11-05 23:28:05 +00:00
John Baldwin
84fea065db Don't modify the destination pointer in ioctl requests.
This breaks the case where the original pointer was NULL but an
in-line IV was used.

Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27064
2020-11-05 23:26:02 +00:00
John Baldwin
9038e6a1e4 Replace some K&R function definitions with ANSI C.
Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27062
2020-11-03 22:32:30 +00:00
John Baldwin
d3d79e968b Consistently use C99 fixed-width types in the in-kernel crypto code.
Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D27061
2020-11-03 22:27:54 +00:00
Mark Johnston
d588dc7d84 opencrypto: Annotate hmac_init_(i|o)pad() to make auth_hash const
This makes them friendlier to drivers that try to use const pointers
whenever possible in their internal structures.

Reviewed by:	jhb
Sponsored by:	Rubicon Communications, LLC (Netgate)
Differential Revision:	https://reviews.freebsd.org/D26901
2020-10-30 17:05:36 +00:00
John Baldwin
e7f6b6cf69 Fix a couple of bugs for asym crypto introduced in r359374.
- Check for null pointers in the crypto_drivers[] array when checking
  for empty slots in crypto_select_kdriver().

- Handle the case where crypto_kdone() is invoked on a request where
  krq_cap is NULL due to not finding a matching driver.

Reviewed by:	markj
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D26811
2020-10-19 20:04:03 +00:00
John Baldwin
ecedef531b Mark asymmetric cryptography via OCF deprecated for 14.0.
Only one MIPS-specific driver implements support for one of the
asymmetric operations.  There are no in-kernel users besides
/dev/crypto.  The only known user of the /dev/crypto interface was the
engine in OpenSSL releases before 1.1.0.  1.1.0 includes a rewritten
engine that does not use the asymmetric operations due to lack of
documentation.

Reviewed by:	cem, markj
MFC after:	1 week
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D26810
2020-10-19 18:21:41 +00:00
Marcin Wojtas
6038018ab1 Add support for ESN in cryptosoft
This patch adds support for IPsec ESN (Extended Sequence Numbers) in
encrypt and authenticate mode (eg. AES-CBC and SHA256) and combined mode
(eg. AES-GCM).

For encrypt and authenticate mode the ESN is stored in separate crp_esn
buffer because the high-order 32 bits of the sequence number are
appended after the Next Header (RFC 4303).

For combined modes the high-order 32 bits of the sequence number [e.g.
RFC 4106, Chapter 5 AAD Construction] are part of crp_aad (prepared by
netipsec layer in case of ESN support enabled), therefore non visible
diff around combined modes.

Submitted by:           Grzegorz Jaszczyk <jaz@semihalf.com>
                        Patryk Duda <pdk@semihalf.com>
Reviewed by:            jhb
Differential revision:  https://reviews.freebsd.org/D22364
Obtained from:          Semihalf
Sponsored by:           Stormshield
2020-10-16 11:18:13 +00:00
Marcin Wojtas
7e89ae49db Prepare crypto framework for IPsec ESN support
This permits requests (netipsec ESP and AH protocol) to provide the
IPsec ESN (Extended Sequence Numbers) in a separate buffer.

As with separate output buffer and separate AAD buffer not all drivers
support this feature. Consumer must request use of this feature via new
session flag.

Submitted by:           Grzegorz Jaszczyk <jaz@semihalf.com>
                        Patryk Duda <pdk@semihalf.com>
Reviewed by:            jhb
Differential revision:  https://reviews.freebsd.org/D24838
Obtained from:          Semihalf
Sponsored by:           Stormshield
2020-10-16 11:06:33 +00:00
John Baldwin
47e2650ea4 Add support to the KTLS OCF module for AES-CBC MTE ciphersuites.
This is a simplistic approach which encrypts each TLS record in two
separate passes: one to generate the MAC and a second to encrypt.
This supports TLS 1.0 connections with implicit IVs as well as TLS
1.1+ with explicit IVs.

Reviewed by:	gallatin
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D26730
2020-10-13 18:04:19 +00:00
John Baldwin
e0b155fe4a Simplify swcr_authcompute() after removal of deprecated algorithms.
- Just use sw->octx != NULL to handle the HMAC case when finalizing
  the MAC.

- Explicitly zero the on-stack auth context.

Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D26688
2020-10-06 18:07:52 +00:00
Warner Losh
2215adc5ff Include sys/types.h here
It's included by header pollution in most of the compile
environments. However, in the standalone envirnment, it's not
included. Go ahead and include it always since the overhead is low and
it is simpler that way.

MFC After: 3 days
2020-09-15 15:21:29 +00:00
John Baldwin
62cddd0e03 Name the on-stack union of compat thunks.
C does not permit an anonymous union at a top-level scope.

Pointy hat to:	jhb
2020-08-26 22:36:08 +00:00
John Baldwin
113bcc82a2 Add freebsd32 compat support for CIOCCRYPTAEAD.
Reviewed by:	markj (earlier version)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D26179
2020-08-26 21:28:47 +00:00
John Baldwin
5612fcb17d Simplify compat shims for /dev/crypto.
- Make session handling always use the CIOGSESSION2 structure.
  CIOGSESSION requests use a thunk similar to COMPAT_FREEBSD32 session
  requests.  This permits the ioctl handler to use the 'crid' field
  unconditionally.

- Move COMPAT_FREEBSD32 handling out of the main ioctl handler body
  and instead do conversions in/out of thunk structures in dedicated
  blocks at the start and end of the ioctl function.

Reviewed by:	markj (earlier version)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D26178
2020-08-26 21:17:18 +00:00
Alan Somers
e6f6d0c9bc crypto(9): add CRYPTO_BUF_VMPAGE
crypto(9) functions can now be used on buffers composed of an array of
vm_page_t structures, such as those stored in an unmapped struct bio.  It
requires the running to kernel to support the direct memory map, so not all
architectures can use it.

Reviewed by:	markj, kib, jhb, mjg, mat, bcr (manpages)
MFC after:	1 week
Sponsored by:	Axcient
Differential Revision:	https://reviews.freebsd.org/D25671
2020-08-26 02:37:42 +00:00
John Baldwin
3c0e568505 Add support for KTLS RX via software decryption.
Allow TLS records to be decrypted in the kernel after being received
by a NIC.  At a high level this is somewhat similar to software KTLS
for the transmit path except in reverse.  Protocols enqueue mbufs
containing encrypted TLS records (or portions of records) into the
tail of a socket buffer and the KTLS layer decrypts those records
before returning them to userland applications.  However, there is an
important difference:

- In the transmit case, the socket buffer is always a single "record"
  holding a chain of mbufs.  Not-yet-encrypted mbufs are marked not
  ready (M_NOTREADY) and released to protocols for transmit by marking
  mbufs ready once their data is encrypted.

- In the receive case, incoming (encrypted) data appended to the
  socket buffer is still a single stream of data from the protocol,
  but decrypted TLS records are stored as separate records in the
  socket buffer and read individually via recvmsg().

Initially I tried to make this work by marking incoming mbufs as
M_NOTREADY, but there didn't seemed to be a non-gross way to deal with
picking a portion of the mbuf chain and turning it into a new record
in the socket buffer after decrypting the TLS record it contained
(along with prepending a control message).  Also, such mbufs would
also need to be "pinned" in some way while they are being decrypted
such that a concurrent sbcut() wouldn't free them out from under the
thread performing decryption.

As such, I settled on the following solution:

- Socket buffers now contain an additional chain of mbufs (sb_mtls,
  sb_mtlstail, and sb_tlscc) containing encrypted mbufs appended by
  the protocol layer.  These mbufs are still marked M_NOTREADY, but
  soreceive*() generally don't know about them (except that they will
  block waiting for data to be decrypted for a blocking read).

- Each time a new mbuf is appended to this TLS mbuf chain, the socket
  buffer peeks at the TLS record header at the head of the chain to
  determine the encrypted record's length.  If enough data is queued
  for the TLS record, the socket is placed on a per-CPU TLS workqueue
  (reusing the existing KTLS workqueues and worker threads).

- The worker thread loops over the TLS mbuf chain decrypting records
  until it runs out of data.  Each record is detached from the TLS
  mbuf chain while it is being decrypted to keep the mbufs "pinned".
  However, a new sb_dtlscc field tracks the character count of the
  detached record and sbcut()/sbdrop() is updated to account for the
  detached record.  After the record is decrypted, the worker thread
  first checks to see if sbcut() dropped the record.  If so, it is
  freed (can happen when a socket is closed with pending data).
  Otherwise, the header and trailer are stripped from the original
  mbufs, a control message is created holding the decrypted TLS
  header, and the decrypted TLS record is appended to the "normal"
  socket buffer chain.

(Side note: the SBCHECK() infrastucture was very useful as I was
 able to add assertions there about the TLS chain that caught several
 bugs during development.)

Tested by:	rmacklem (various versions)
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24628
2020-07-23 23:48:18 +00:00
John Baldwin
70d1a4351a Consolidate duplicated code into a ktls_ocf_dispatch function.
This function manages the loop around crypto_dispatch and coordination
with ktls_ocf_callback.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D25757
2020-07-23 21:43:06 +00:00
John Baldwin
33a1a488d5 Don't dynamically allocate data structures for KTLS crypto requests.
Allocate iovec arrays and struct cryptop and struct ocf_operation
objects on the stack to reduce avoid the overhead of malloc().

These structures are all small enough to fit on the stack of the KTLS
worker threads.

Reviewed by:	gallatin
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D25692
2020-07-20 22:32:39 +00:00