Commit Graph

255 Commits

Author SHA1 Message Date
Warner Losh
2215adc5ff Include sys/types.h here
It's included by header pollution in most of the compile
environments. However, in the standalone envirnment, it's not
included. Go ahead and include it always since the overhead is low and
it is simpler that way.

MFC After: 3 days
2020-09-15 15:21:29 +00:00
John Baldwin
62cddd0e03 Name the on-stack union of compat thunks.
C does not permit an anonymous union at a top-level scope.

Pointy hat to:	jhb
2020-08-26 22:36:08 +00:00
John Baldwin
113bcc82a2 Add freebsd32 compat support for CIOCCRYPTAEAD.
Reviewed by:	markj (earlier version)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D26179
2020-08-26 21:28:47 +00:00
John Baldwin
5612fcb17d Simplify compat shims for /dev/crypto.
- Make session handling always use the CIOGSESSION2 structure.
  CIOGSESSION requests use a thunk similar to COMPAT_FREEBSD32 session
  requests.  This permits the ioctl handler to use the 'crid' field
  unconditionally.

- Move COMPAT_FREEBSD32 handling out of the main ioctl handler body
  and instead do conversions in/out of thunk structures in dedicated
  blocks at the start and end of the ioctl function.

Reviewed by:	markj (earlier version)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D26178
2020-08-26 21:17:18 +00:00
Alan Somers
e6f6d0c9bc crypto(9): add CRYPTO_BUF_VMPAGE
crypto(9) functions can now be used on buffers composed of an array of
vm_page_t structures, such as those stored in an unmapped struct bio.  It
requires the running to kernel to support the direct memory map, so not all
architectures can use it.

Reviewed by:	markj, kib, jhb, mjg, mat, bcr (manpages)
MFC after:	1 week
Sponsored by:	Axcient
Differential Revision:	https://reviews.freebsd.org/D25671
2020-08-26 02:37:42 +00:00
John Baldwin
3c0e568505 Add support for KTLS RX via software decryption.
Allow TLS records to be decrypted in the kernel after being received
by a NIC.  At a high level this is somewhat similar to software KTLS
for the transmit path except in reverse.  Protocols enqueue mbufs
containing encrypted TLS records (or portions of records) into the
tail of a socket buffer and the KTLS layer decrypts those records
before returning them to userland applications.  However, there is an
important difference:

- In the transmit case, the socket buffer is always a single "record"
  holding a chain of mbufs.  Not-yet-encrypted mbufs are marked not
  ready (M_NOTREADY) and released to protocols for transmit by marking
  mbufs ready once their data is encrypted.

- In the receive case, incoming (encrypted) data appended to the
  socket buffer is still a single stream of data from the protocol,
  but decrypted TLS records are stored as separate records in the
  socket buffer and read individually via recvmsg().

Initially I tried to make this work by marking incoming mbufs as
M_NOTREADY, but there didn't seemed to be a non-gross way to deal with
picking a portion of the mbuf chain and turning it into a new record
in the socket buffer after decrypting the TLS record it contained
(along with prepending a control message).  Also, such mbufs would
also need to be "pinned" in some way while they are being decrypted
such that a concurrent sbcut() wouldn't free them out from under the
thread performing decryption.

As such, I settled on the following solution:

- Socket buffers now contain an additional chain of mbufs (sb_mtls,
  sb_mtlstail, and sb_tlscc) containing encrypted mbufs appended by
  the protocol layer.  These mbufs are still marked M_NOTREADY, but
  soreceive*() generally don't know about them (except that they will
  block waiting for data to be decrypted for a blocking read).

- Each time a new mbuf is appended to this TLS mbuf chain, the socket
  buffer peeks at the TLS record header at the head of the chain to
  determine the encrypted record's length.  If enough data is queued
  for the TLS record, the socket is placed on a per-CPU TLS workqueue
  (reusing the existing KTLS workqueues and worker threads).

- The worker thread loops over the TLS mbuf chain decrypting records
  until it runs out of data.  Each record is detached from the TLS
  mbuf chain while it is being decrypted to keep the mbufs "pinned".
  However, a new sb_dtlscc field tracks the character count of the
  detached record and sbcut()/sbdrop() is updated to account for the
  detached record.  After the record is decrypted, the worker thread
  first checks to see if sbcut() dropped the record.  If so, it is
  freed (can happen when a socket is closed with pending data).
  Otherwise, the header and trailer are stripped from the original
  mbufs, a control message is created holding the decrypted TLS
  header, and the decrypted TLS record is appended to the "normal"
  socket buffer chain.

(Side note: the SBCHECK() infrastucture was very useful as I was
 able to add assertions there about the TLS chain that caught several
 bugs during development.)

Tested by:	rmacklem (various versions)
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24628
2020-07-23 23:48:18 +00:00
John Baldwin
70d1a4351a Consolidate duplicated code into a ktls_ocf_dispatch function.
This function manages the loop around crypto_dispatch and coordination
with ktls_ocf_callback.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D25757
2020-07-23 21:43:06 +00:00
John Baldwin
33a1a488d5 Don't dynamically allocate data structures for KTLS crypto requests.
Allocate iovec arrays and struct cryptop and struct ocf_operation
objects on the stack to reduce avoid the overhead of malloc().

These structures are all small enough to fit on the stack of the KTLS
worker threads.

Reviewed by:	gallatin
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D25692
2020-07-20 22:32:39 +00:00
Mark Johnston
e1a82b35bf crypto(9): Stop checking for failures from malloc(M_WAITOK).
PR:		240545
Submitted by:	Andrew Reiter <arr@watson.org>
Reviewed by:	cem, delphij, jhb
MFC after:	1 week
Event:		July 2020 Bugathon
2020-07-20 17:44:13 +00:00
Mark Johnston
e5587cbbc2 Clean up crypto_init().
The function is called from a KLD load handler, so it may sleep.

- Stop checking for errors from uma_zcreate(), they don't happen.
- Convert M_NOWAIT allocations to M_WAITOK.
- Remove error handling for existing M_WAITOK allocations.
- Fix style.

Reviewed by:	cem, delphij, jhb
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D25696
2020-07-17 14:45:16 +00:00
John Baldwin
946b8f6fb0 Add crypto_initreq() and crypto_destroyreq().
These routines are similar to crypto_getreq() and crypto_freereq() but
operate on caller-supplied storage instead of allocating crypto
requests from a UMA zone.

Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D25691
2020-07-16 21:30:46 +00:00
Mark Johnston
7290cb47fc Convert cryptostats to a counter_u64 array.
The global counters were not SMP-friendly.  Use per-CPU counters
instead.

Reviewed by:	jhb
Sponsored by:	Rubicon Communications, LLC (Netgate)
Differential Revision:	https://reviews.freebsd.org/D25466
2020-06-30 22:01:21 +00:00
Mark Johnston
a5ae70f5a0 Remove unused 32-bit compatibility structures from cryptodev.
The counters are exported by a sysctl and have the same width on all
platforms anyway.

Reviewed by:	cem, delphij, jhb
Sponsored by:	Rubicon Communications, LLC (Netgate)
Differential Revision:	https://reviews.freebsd.org/D25465
2020-06-30 15:57:11 +00:00
Mark Johnston
a5c053f5a7 Remove CRYPTO_TIMING.
It was added a very long time ago.  It is single-threaded, so only
really useful for basic measurements, and in the meantime we've gotten
some more sophisticated profiling tools.

Reviewed by:	cem, delphij, jhb
Sponsored by:	Rubicon Communications, LLC (Netgate)
Differential Revision:	https://reviews.freebsd.org/D25464
2020-06-30 15:56:54 +00:00
John Baldwin
17a831ea25 Zero the temporary HMAC key in hmac_init_pad().
Reviewed by:	delphij
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D25436
2020-06-25 20:18:55 +00:00
John Baldwin
4a711b8d04 Use zfree() instead of explicit_bzero() and free().
In addition to reducing lines of code, this also ensures that the full
allocation is always zeroed avoiding possible bugs with incorrect
lengths passed to explicit_bzero().

Suggested by:	cem
Reviewed by:	cem, delphij
Approved by:	csprng (cem)
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D25435
2020-06-25 20:17:34 +00:00
John Baldwin
5b750b9a68 Store the AAD in a separate buffer for KTLS.
For TLS 1.2 this permits reusing one of the existing iovecs without
always having to duplicate both.

While here, only duplicate the output iovec for TLS 1.3 if it will be
used.

Reviewed by:	gallatin
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D25291
2020-06-23 00:02:28 +00:00
John Baldwin
9b774dc0c5 Add support to the crypto framework for separate AAD buffers.
This permits requests to provide the AAD in a separate side buffer
instead of as a region in the crypto request input buffer.  This is
useful when the main data buffer might not contain the full AAD
(e.g. for TLS or IPsec with ESN).

Unlike separate IVs which are constrained in size and stored in an
array in struct cryptop, separate AAD is provided by the caller
setting a new crp_aad pointer to the buffer.  The caller must ensure
the pointer remains valid and the buffer contents static until the
request is completed (e.g. when the callback routine is invoked).

As with separate output buffers, not all drivers support this feature.
Consumers must request use of this feature via a new session flag.

To aid in driver testing, kern.crypto.cryptodev_separate_aad can be
set to force /dev/crypto requests to use a separate AAD buffer.

Discussed with:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D25288
2020-06-22 23:20:43 +00:00
John Baldwin
26d292d3e2 Various optimizations to software AES-CCM and AES-GCM.
- Make use of cursors to avoid data copies for AES-CCM and AES-GCM.

  Pass pointers into the request's input and/or output buffers
  directly to the Update, encrypt, and decrypt hooks rather than
  always copying all data into a temporary block buffer on the stack.

- Move handling for partial final blocks out of the main loop.

  This removes branches from the main loop and permits using
  encrypt/decrypt_last which avoids a memset to clear the rest of the
  block on the stack.

- Shrink the on-stack buffers to assume AES block sizes and CCM/GCM
  tag lengths.

- For AAD data, pass larger chunks to axf->Update.  CCM can take each
  AAD segment in a single call.  GMAC can take multiple blocks at a
  time.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D25058
2020-06-12 23:10:30 +00:00
John Baldwin
4e6a381306 Fix a regression in r361804 for TLS 1.3.
I was not including the record type stored in the first byte of the
trailer as part of the payload to be encrypted and hashed.

Sponsored by:	Netflix
2020-06-12 22:27:26 +00:00
John Baldwin
9b6b2f8608 Adjust crypto_apply function callbacks for OCF.
- crypto_apply() is only used for reading a buffer to compute a
  digest, so change the data pointer to a const pointer.

- To better match m_apply(), change the data pointer type to void *
  and the length from uint16_t to u_int.  The length field in
  particular matters as none of the apply logic was splitting requests
  larger than UINT16_MAX.

- Adjust the auth_xform Update callback to match the function
  prototype passed to crypto_apply() and crypto_apply_buf().  This
  removes the needs for casts when using the Update callback.

- Change the Reinit and Setkey callbacks to also use a u_int length
  instead of uint16_t.

- Update auth transforms for the changes.  While here, use C99
  initializers for auth_hash structures and avoid casts on callbacks.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D25171
2020-06-10 21:18:19 +00:00
John Baldwin
a3d565a118 Add a crypto capability flag for accelerated software drivers.
Use this in GELI to print out a different message when accelerated
software such as AESNI is used vs plain software crypto.

While here, simplify the logic in GELI a bit for determing which type
of crypto driver was chosen the first time by examining the
capabilities of the matched driver after a single call to
crypto_newsession rather than making separate calls with different
flags.

Reviewed by:	delphij
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D25126
2020-06-09 22:26:07 +00:00
John Baldwin
080933c050 Use separate output buffers for OCF requests in KTLS.
KTLS encryption requests for file-backed data such as from sendfile(2)
require the encrypted data to be stored in a separate buffer from the
unencrypted file input data.  Previously the OCF backend for KTLS
manually copied the data from the input buffer to the output buffer
before queueing the crypto request.  Now the OCF backend will use a
separate output buffer for such requests and avoid the copy.  This
mostly helps when an async co-processor is used by saving CPU cycles
used on the copy.

Reviewed by:	gallatin (earlier version)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24545
2020-06-04 20:12:34 +00:00
John Baldwin
20c128da91 Add explicit bzero's of sensitive data in software crypto consumers.
Explicitly zero IVs, block buffers, and hashes/digests.

Reviewed by:	delphij
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D25057
2020-06-03 22:11:05 +00:00
John Baldwin
4542cd9379 Increment the correct pointer when a crypto buffer spans an mbuf or iovec.
When a crypto_cursor_copyback() request spanned multiple mbufs or
iovecs, the pointer into the mbuf/iovec was incremented instead of the
pointer into the source buffer being copied from.

PR:		246737
Reported by:	Jenkins, ZFS test suite
Sponsored by:	Netflix
2020-05-29 05:41:21 +00:00
John Baldwin
ba63e5e701 Add a sysctl knob to use separate output buffers for /dev/crypto.
This is a testing aid to permit using testing a driver's support of
separate output buffers via cryptocheck.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24545
2020-05-25 22:21:09 +00:00
John Baldwin
33f3bad35e Export the _kern_crypto sysctl node from crypto.c.
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24545
2020-05-25 22:18:33 +00:00
John Baldwin
9c0e3d3a53 Add support for optional separate output buffers to in-kernel crypto.
Some crypto consumers such as GELI and KTLS for file-backed sendfile
need to store their output in a separate buffer from the input.
Currently these consumers copy the contents of the input buffer into
the output buffer and queue an in-place crypto operation on the output
buffer.  Using a separate output buffer avoids this copy.

- Create a new 'struct crypto_buffer' describing a crypto buffer
  containing a type and type-specific fields.  crp_ilen is gone,
  instead buffers that use a flat kernel buffer have a cb_buf_len
  field for their length.  The length of other buffer types is
  inferred from the backing store (e.g. uio_resid for a uio).
  Requests now have two such structures: crp_buf for the input buffer,
  and crp_obuf for the output buffer.

- Consumers now use helper functions (crypto_use_*,
  e.g. crypto_use_mbuf()) to configure the input buffer.  If an output
  buffer is not configured, the request still modifies the input
  buffer in-place.  A consumer uses a second set of helper functions
  (crypto_use_output_*) to configure an output buffer.

- Consumers must request support for separate output buffers when
  creating a crypto session via the CSP_F_SEPARATE_OUTPUT flag and are
  only permitted to queue a request with a separate output buffer on
  sessions with this flag set.  Existing drivers already reject
  sessions with unknown flags, so this permits drivers to be modified
  to support this extension without requiring all drivers to change.

- Several data-related functions now have matching versions that
  operate on an explicit buffer (e.g. crypto_apply_buf,
  crypto_contiguous_subsegment_buf, bus_dma_load_crp_buf).

- Most of the existing data-related functions operate on the input
  buffer.  However crypto_copyback always writes to the output buffer
  if a request uses a separate output buffer.

- For the regions in input/output buffers, the following conventions
  are followed:
  - AAD and IV are always present in input only and their
    fields are offsets into the input buffer.
  - payload is always present in both buffers.  If a request uses a
    separate output buffer, it must set a new crp_payload_start_output
    field to the offset of the payload in the output buffer.
  - digest is in the input buffer for verify operations, and in the
    output buffer for compute operations.  crp_digest_start is relative
    to the appropriate buffer.

- Add a crypto buffer cursor abstraction.  This is a more general form
  of some bits in the cryptosoft driver that tried to always use uio's.
  However, compared to the original code, this avoids rewalking the uio
  iovec array for requests with multiple vectors.  It also avoids
  allocate an iovec array for mbufs and populating it by instead walking
  the mbuf chain directly.

- Update the cryptosoft(4) driver to support separate output buffers
  making use of the cursor abstraction.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24545
2020-05-25 22:12:04 +00:00
John Baldwin
d65771e472 Correct the minimum key length for Camellia to 16 bytes (128 bits).
MFC after:	1 week
2020-05-22 17:21:22 +00:00
John Baldwin
723d87648e Improve support for stream ciphers in the software encryption interface.
Add a 'native_blocksize' member to 'struct enc_xform' that ciphers can
use if they support a partial final block.  This is particular useful
for stream ciphers, but can also apply to other ciphers.  cryptosoft
will only pass in native blocks to the encrypt and decrypt hooks.  For
the final partial block, 'struct enc_xform' now has new
encrypt_last/decrypt_last hooks which accept the length of the final
block.  The multi_block methods are also retired.

Mark AES-ICM (AES-CTR) as a stream cipher.  This has some interesting
effects on IPsec in that FreeBSD can now properly receive all packets
sent by Linux when using AES-CTR, but FreeBSD can no longer
interoperate with OpenBSD and older verisons of FreeBSD which assume
AES-CTR packets have a payload padded to a 16-byte boundary.  Kornel
has offered to work on a patch to add a compatiblity sysctl to enforce
additional padding for AES-CTR in esp_output to permit compatibility
with OpenBSD and older versions of FreeBSD.

AES-XTS continues to use a block size of a single AES block length.
It is possible to adjust it to support partial final blocks by
implementing cipher text stealing via encrypt_last/decrypt_last hooks,
but I have not done so.

Reviewed by:	cem (earlier version)
Tested by:	Kornel Dulęba <mindal@semihalf.com> (AES-CTR with IPsec)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24906
2020-05-22 16:29:09 +00:00
John Baldwin
4f98ffdd1d Fix libstand build breakage after r361298.
- Use enc_xform_aes_xts.setkey() directly instead of duplicating the code
  now that it no longer calls malloc().
- Rather than bringing back all of xform_userland.h, add a conditional
  #include of <stand.h> to xform_enc.h.
- Update calls to encrypt/decrypt callbacks in enc_xform_aes_xts for
  separate input/output pointers.

Pointy hat to:	jhb
2020-05-20 22:25:41 +00:00
John Baldwin
3e9470482a Various cleanups to the software encryption transform interface.
- Consistently use 'void *' for key schedules / key contexts instead
  of a mix of 'caddr_t', 'uint8_t *', and 'void *'.

- Add a ctxsize member to enc_xform similar to what auth transforms use
  and require callers to malloc/zfree the context.  The setkey callback
  now supplies the caller-allocated context pointer and the zerokey
  callback is removed.  Callers now always use zfree() to ensure
  key contexts are zeroed.

- Consistently use C99 initializers for all statically-initialized
  instances of 'struct enc_xform'.

- Change the encrypt and decrypt functions to accept separate in and
  out buffer pointers.  Almost all of the backend crypto functions
  already supported separate input and output buffers and this makes
  it simpler to support separate buffers in OCF.

- Remove xform_userland.h shim to permit transforms to be compiled in
  userland.  Transforms no longer call malloc/free directly.

Reviewed by:	cem (earlier version)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24855
2020-05-20 21:21:01 +00:00
John Baldwin
f272bc03cc Trim a few more things I missed from xform_enc.h.
An extern declaration for the now-removed Blowfish encryption
transform, and an include of the DES header.
2020-05-13 18:36:02 +00:00
John Baldwin
63823cac92 Remove MD5 HMAC from OCF.
There are no in-kernel consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24775
2020-05-11 22:08:08 +00:00
Kristof Provost
38dc0eac57 opencrypto: Add missing ioctl exit SDTs
The opencrypto ioctl code has very useful probe points at the various exit
points. These allow us to figure out exactly why a request failed. However, a
few paths did not have these probe points. Add them here.

Reviewed by:	jhb
2020-05-11 21:42:19 +00:00
John Baldwin
0e00c709d7 Remove support for DES and Triple DES from OCF.
It no longer has any in-kernel consumers via OCF.  smbfs still uses
single DES directly, so sys/crypto/des remains for that use case.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24773
2020-05-11 21:34:29 +00:00
John Baldwin
32075647ef Remove support for the Blowfish algorithm from OCF.
It no longer has any in-kernel consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24772
2020-05-11 21:24:05 +00:00
John Baldwin
33fb013e16 Remove support for the ARC4 algorithm from OCF.
There are no longer any in-kernel consumers.  The software
implementation was also a non-functional stub.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24771
2020-05-11 21:17:08 +00:00
John Baldwin
3a0b6a93a7 Remove support for keyed MD5 and SHA1 authentication hashes.
They no longer have any in-tree consumers.  Note that these are a
different from MD5-HMAC and SHA1-HMAC and were only used with IPsec.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24770
2020-05-11 21:04:59 +00:00
John Baldwin
5e46d47f93 Remove support for the skipjack encryption algorithm.
This was removed from IPsec in r286100 and no longer has any in-tree
consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24769
2020-05-11 20:54:59 +00:00
John Baldwin
7971a6f911 Remove support for the cast128 encryption algorithm.
It no longer has any in-tree consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24768
2020-05-11 20:52:18 +00:00
John Baldwin
9b5631807e Remove incomplete support for plain MD5 from OCF.
Although a few drivers supported this algorithm, there were never any
in-kernel consumers.  cryptosoft and cryptodev never supported it,
and there was not a software xform auth_hash for it.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24767
2020-05-11 20:40:30 +00:00
John Baldwin
ad5570559b Use a lookup table of algorithm types for alg_is_* helpers.
This makes it easier to maintain these functions as algorithms are
added or removed.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24668
2020-05-04 22:31:38 +00:00
John Baldwin
62e107c31b Actually remove support for Triple DES, not just the warning.
Missed in r360562.
2020-05-02 17:18:32 +00:00
John Baldwin
6c80c319ef Remove support for the algorithms deprecated in r348876.
This removes support for the following algorithms:
- ARC4
- Blowfish
- CAST128
- DES
- 3DES
- MD5-HMAC
- Skipjack

Since /dev/crypto no longer supports 3DES, stop testing the 3DES KAT
vectors in cryptotest.py.

Reviewed by:	cem (previous version)
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24346
2020-05-02 14:20:32 +00:00
John Baldwin
29fe41ddd7 Retire the CRYPTO_F_IV_GENERATE flag.
The sole in-tree user of this flag has been retired, so remove this
complexity from all drivers.  While here, add a helper routine drivers
can use to read the current request's IV into a local buffer.  Use
this routine to replace duplicated code in nearly all drivers.

Reviewed by:	cem
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24450
2020-04-20 22:24:49 +00:00
John Baldwin
1b96435a65 Avoid checking pointers that are never NULL.
Coverity noted that cod pointer is always non-NULL at the end of
cryptodev_aead().  While here, fix cryptodev_op() to match by making
one earlier failure case before cod and crp are allocated just return
directly.

CID:		1422185
Reported by:	Coverity
2020-04-02 17:22:16 +00:00
John Baldwin
54ae1f3308 Remove duplicate conditional.
CID:		1422192
Reported by:	Coverity
2020-04-02 16:55:28 +00:00
John Baldwin
c034143269 Refactor driver and consumer interfaces for OCF (in-kernel crypto).
- The linked list of cryptoini structures used in session
  initialization is replaced with a new flat structure: struct
  crypto_session_params.  This session includes a new mode to define
  how the other fields should be interpreted.  Available modes
  include:

  - COMPRESS (for compression/decompression)
  - CIPHER (for simply encryption/decryption)
  - DIGEST (computing and verifying digests)
  - AEAD (combined auth and encryption such as AES-GCM and AES-CCM)
  - ETA (combined auth and encryption using encrypt-then-authenticate)

  Additional modes could be added in the future (e.g. if we wanted to
  support TLS MtE for AES-CBC in the kernel we could add a new mode
  for that.  TLS modes might also affect how AAD is interpreted, etc.)

  The flat structure also includes the key lengths and algorithms as
  before.  However, code doesn't have to walk the linked list and
  switch on the algorithm to determine which key is the auth key vs
  encryption key.  The 'csp_auth_*' fields are always used for auth
  keys and settings and 'csp_cipher_*' for cipher.  (Compression
  algorithms are stored in csp_cipher_alg.)

- Drivers no longer register a list of supported algorithms.  This
  doesn't quite work when you factor in modes (e.g. a driver might
  support both AES-CBC and SHA2-256-HMAC separately but not combined
  for ETA).  Instead, a new 'crypto_probesession' method has been
  added to the kobj interface for symmteric crypto drivers.  This
  method returns a negative value on success (similar to how
  device_probe works) and the crypto framework uses this value to pick
  the "best" driver.  There are three constants for hardware
  (e.g. ccr), accelerated software (e.g. aesni), and plain software
  (cryptosoft) that give preference in that order.  One effect of this
  is that if you request only hardware when creating a new session,
  you will no longer get a session using accelerated software.
  Another effect is that the default setting to disallow software
  crypto via /dev/crypto now disables accelerated software.

  Once a driver is chosen, 'crypto_newsession' is invoked as before.

- Crypto operations are now solely described by the flat 'cryptop'
  structure.  The linked list of descriptors has been removed.

  A separate enum has been added to describe the type of data buffer
  in use instead of using CRYPTO_F_* flags to make it easier to add
  more types in the future if needed (e.g. wired userspace buffers for
  zero-copy).  It will also make it easier to re-introduce separate
  input and output buffers (in-kernel TLS would benefit from this).

  Try to make the flags related to IV handling less insane:

  - CRYPTO_F_IV_SEPARATE means that the IV is stored in the 'crp_iv'
    member of the operation structure.  If this flag is not set, the
    IV is stored in the data buffer at the 'crp_iv_start' offset.

  - CRYPTO_F_IV_GENERATE means that a random IV should be generated
    and stored into the data buffer.  This cannot be used with
    CRYPTO_F_IV_SEPARATE.

  If a consumer wants to deal with explicit vs implicit IVs, etc. it
  can always generate the IV however it needs and store partial IVs in
  the buffer and the full IV/nonce in crp_iv and set
  CRYPTO_F_IV_SEPARATE.

  The layout of the buffer is now described via fields in cryptop.
  crp_aad_start and crp_aad_length define the boundaries of any AAD.
  Previously with GCM and CCM you defined an auth crd with this range,
  but for ETA your auth crd had to span both the AAD and plaintext
  (and they had to be adjacent).

  crp_payload_start and crp_payload_length define the boundaries of
  the plaintext/ciphertext.  Modes that only do a single operation
  (COMPRESS, CIPHER, DIGEST) should only use this region and leave the
  AAD region empty.

  If a digest is present (or should be generated), it's starting
  location is marked by crp_digest_start.

  Instead of using the CRD_F_ENCRYPT flag to determine the direction
  of the operation, cryptop now includes an 'op' field defining the
  operation to perform.  For digests I've added a new VERIFY digest
  mode which assumes a digest is present in the input and fails the
  request with EBADMSG if it doesn't match the internally-computed
  digest.  GCM and CCM already assumed this, and the new AEAD mode
  requires this for decryption.  The new ETA mode now also requires
  this for decryption, so IPsec and GELI no longer do their own
  authentication verification.  Simple DIGEST operations can also do
  this, though there are no in-tree consumers.

  To eventually support some refcounting to close races, the session
  cookie is now passed to crypto_getop() and clients should no longer
  set crp_sesssion directly.

- Assymteric crypto operation structures should be allocated via
  crypto_getkreq() and freed via crypto_freekreq().  This permits the
  crypto layer to track open asym requests and close races with a
  driver trying to unregister while asym requests are in flight.

- crypto_copyback, crypto_copydata, crypto_apply, and
  crypto_contiguous_subsegment now accept the 'crp' object as the
  first parameter instead of individual members.  This makes it easier
  to deal with different buffer types in the future as well as
  separate input and output buffers.  It's also simpler for driver
  writers to use.

- bus_dmamap_load_crp() loads a DMA mapping for a crypto buffer.
  This understands the various types of buffers so that drivers that
  use DMA do not have to be aware of different buffer types.

- Helper routines now exist to build an auth context for HMAC IPAD
  and OPAD.  This reduces some duplicated work among drivers.

- Key buffers are now treated as const throughout the framework and in
  device drivers.  However, session key buffers provided when a session
  is created are expected to remain alive for the duration of the
  session.

- GCM and CCM sessions now only specify a cipher algorithm and a cipher
  key.  The redundant auth information is not needed or used.

- For cryptosoft, split up the code a bit such that the 'process'
  callback now invokes a function pointer in the session.  This
  function pointer is set based on the mode (in effect) though it
  simplifies a few edge cases that would otherwise be in the switch in
  'process'.

  It does split up GCM vs CCM which I think is more readable even if there
  is some duplication.

- I changed /dev/crypto to support GMAC requests using CRYPTO_AES_NIST_GMAC
  as an auth algorithm and updated cryptocheck to work with it.

- Combined cipher and auth sessions via /dev/crypto now always use ETA
  mode.  The COP_F_CIPHER_FIRST flag is now a no-op that is ignored.
  This was actually documented as being true in crypto(4) before, but
  the code had not implemented this before I added the CIPHER_FIRST
  flag.

- I have not yet updated /dev/crypto to be aware of explicit modes for
  sessions.  I will probably do that at some point in the future as well
  as teach it about IV/nonce and tag lengths for AEAD so we can support
  all of the NIST KAT tests for GCM and CCM.

- I've split up the exising crypto.9 manpage into several pages
  of which many are written from scratch.

- I have converted all drivers and consumers in the tree and verified
  that they compile, but I have not tested all of them.  I have tested
  the following drivers:

  - cryptosoft
  - aesni (AES only)
  - blake2
  - ccr

  and the following consumers:

  - cryptodev
  - IPsec
  - ktls_ocf
  - GELI (lightly)

  I have not tested the following:

  - ccp
  - aesni with sha
  - hifn
  - kgssapi_krb5
  - ubsec
  - padlock
  - safe
  - armv8_crypto (aarch64)
  - glxsb (i386)
  - sec (ppc)
  - cesa (armv7)
  - cryptocteon (mips64)
  - nlmsec (mips64)

Discussed with:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D23677
2020-03-27 18:25:23 +00:00
Pawel Biernacki
7029da5c36 Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)
r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE.  All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by:	kib (mentor, blanket)
Commented by:	kib, gallatin, melifaro
Differential Revision:	https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00