Once a crypto cursor has reached the end of its buffer, it is invalid to
call crypto_cursor_segment() for at least some crypto buffer types.
Reorganize loops to avoid this.
Fixes: cfb7b942be ("cryptosoft: Use multi-block encrypt/decrypt for non-AEAD ciphers.")
Fixes: a221a8f4a0 ("cryptosoft: Use multi-block encrypt/decrypt for AES-GCM.")
Fixes: f8580fcaa1 ("cryptosoft: Use multi-block encrypt/decrypt for AES-CCM.")
Fixes: 5022c68732 ("cryptosoft: Use multi-block encrypt/decrypt for ChaCha20-Poly1305.")
Reported and tested by: madpilot
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
This better reflects the variables purpose and matches other functions
in this file.
Requested by: markj
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33755
This cipher is a wrapper around the ChaCha20-Poly1305 AEAD cipher
which accepts a larger nonce. Part of the nonce is used along with
the key as an input to HChaCha20 to generate a derived key used for
ChaCha20-Poly1305.
This cipher is used by WireGuard.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33523
Group sensitive on-stack variables into anonymous structs so that they
can be cleared with a single call to explicit_bzero rather than
multiple calls.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33527
Reusing the storage removes the need for an additional explicit_bzero.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33526
All of the CBC ciphers now handle block chaining in the enc_xform.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33489
Despite it's name, CRYPTO_NULL_HMAC is actually an unkeyed digest
similar to plain SHA-1 rather than a keyed HMAC algorithm.
Reported by: syzbot+17d7c656f1a797b0faae@syzkaller.appspotmail.com
Discussed with: markj
Fixes: 6113a08b98 cryptosoft: Fully support per-operation keys for auth algorithms.
Sponsored by: The FreeBSD Foundation
All of the request handlers no longer modify session state, so remove
the mutex limiting operations to one per session. In addition, change
the pointer to the session state passed to process callbacks to const.
Suggested by: mjg
Reviewed by: mjg, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33317
Only pre-allocate auth contexts when a session-wide key is provided or
for sessions without keys. For sessions with per-operation keys,
always initialize the on-stack context directly rather than
initializing the session context in swcr_authprepare (now removed) and
then copying that session context into the on-stack context.
This approach permits parallel auth operations without needing a
serializing lock. In addition, the previous code assumed that auth
sessions always provided an initial key unlike cipher sessions which
assume either an initial key or per-op keys.
While here, fix the Blake2 auth transforms to function like other auth
transforms where Setkey is invoked after Init rather than before.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33316
As is done with authentication contexts, allocate cipher contexts on
the stack while completing requests. This permits safely dispatching
concurrent requests on a single session. The cipher context in the
session is now only allocated when a session key is provided during
session setup to serve as a template to initialize the on-stack
context similar to auth operations.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33198
The cipher context isn't always a key schedule, so use a more generic
name.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33197
Extend struct enc_xform to add new members to handle auth operations
for AEAD ciphers. In particular, AEAD operations in cryptosoft no
longer use a struct auth_hash. Instead, the setkey and reinit methods
of struct enc_xform are responsible for initializing both the cipher
and auth state.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33196
This centralizes the check for valid nonce lengths for AES-GCM.
While here, remove some duplicate checks for valid AES-GCM tag lengths
from ccp(4) and ccr(4).
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33194
This is useful for WireGuard which uses a nonce of 8 bytes rather
than the 12 bytes used for IPsec and TLS.
Note that this also fixes a (should be) harmless bug in ossl(4) where
the counter was incorrectly treated as a 64-bit counter instead of a
32-bit counter in terms of wrapping when using a 12 byte nonce.
However, this required a single message (TLS record) longer than 64 *
(2^32 - 1) bytes (about 256 GB) to trigger.
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32122
The tag length is included as one of the values in the flags byte of
block 0 passed to CBC_MAC, so merely copying the first N bytes is
insufficient.
To avoid adding more sideband data to the CBC MAC software context,
pull the generation of block 0, the AAD length, and AAD padding out of
cbc_mac.c and into cryptosoft.c. This matches how GCM/GMAC are
handled where the length block is constructed in cryptosoft.c and
passed as an input to the Update callback. As a result, the CBC MAC
Update() routine is now much simpler and simply performs the
XOR-and-encrypt step on each input block.
While here, avoid a copy to the staging block in the Update routine
when one or more full blocks are passed as input to the Update
callback.
Reviewed by: sef
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32120
Permit nonces of lengths 7 through 13 in the OCF framework and the
cryptosoft driver. A helper function (ccm_max_payload_length) can be
used in OCF drivers to reject CCM requests which are too large for the
specified nonce length.
Reviewed by: sef
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32111
Rather than copying crp_iv to a local array on the stack that is then
passed to xform reinit routines, pass crp_iv directly and remove the
local copy.
Reviewed by: markj
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32106
Add a 'len' argument to the reinit hook in 'struct enc_xform' to
permit support for AEAD ciphers such as AES-CCM and Chacha20-Poly1305
which support different nonce lengths.
Reviewed by: markj
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32105
This is intended for use in KTLS transmit where each TLS record is
described by a single mbuf that is itself queued in the socket buffer.
Using the existing CRYPTO_BUF_MBUF would result in
bus_dmamap_load_crp() walking additional mbufs in the socket buffer
that are not relevant, but generating a S/G list that potentially
exceeds the limit of the tag (while also wasting CPU cycles).
Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30136
cryptosoft is always present and doesn't print any useful information
when it attaches.
Reviewed by: jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29098
This uses the chacha20 IETF and poly1305 implementations from
libsodium. A seperate auth_hash is created for the auth side whose
Setkey method derives the poly1305 key from the AEAD key and nonce as
described in RFC 8439.
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D27837
This patch adds support for IPsec ESN (Extended Sequence Numbers) in
encrypt and authenticate mode (eg. AES-CBC and SHA256) and combined mode
(eg. AES-GCM).
For encrypt and authenticate mode the ESN is stored in separate crp_esn
buffer because the high-order 32 bits of the sequence number are
appended after the Next Header (RFC 4303).
For combined modes the high-order 32 bits of the sequence number [e.g.
RFC 4106, Chapter 5 AAD Construction] are part of crp_aad (prepared by
netipsec layer in case of ESN support enabled), therefore non visible
diff around combined modes.
Submitted by: Grzegorz Jaszczyk <jaz@semihalf.com>
Patryk Duda <pdk@semihalf.com>
Reviewed by: jhb
Differential revision: https://reviews.freebsd.org/D22364
Obtained from: Semihalf
Sponsored by: Stormshield
- Just use sw->octx != NULL to handle the HMAC case when finalizing
the MAC.
- Explicitly zero the on-stack auth context.
Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D26688
crypto(9) functions can now be used on buffers composed of an array of
vm_page_t structures, such as those stored in an unmapped struct bio. It
requires the running to kernel to support the direct memory map, so not all
architectures can use it.
Reviewed by: markj, kib, jhb, mjg, mat, bcr (manpages)
MFC after: 1 week
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D25671
In addition to reducing lines of code, this also ensures that the full
allocation is always zeroed avoiding possible bugs with incorrect
lengths passed to explicit_bzero().
Suggested by: cem
Reviewed by: cem, delphij
Approved by: csprng (cem)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25435
This permits requests to provide the AAD in a separate side buffer
instead of as a region in the crypto request input buffer. This is
useful when the main data buffer might not contain the full AAD
(e.g. for TLS or IPsec with ESN).
Unlike separate IVs which are constrained in size and stored in an
array in struct cryptop, separate AAD is provided by the caller
setting a new crp_aad pointer to the buffer. The caller must ensure
the pointer remains valid and the buffer contents static until the
request is completed (e.g. when the callback routine is invoked).
As with separate output buffers, not all drivers support this feature.
Consumers must request use of this feature via a new session flag.
To aid in driver testing, kern.crypto.cryptodev_separate_aad can be
set to force /dev/crypto requests to use a separate AAD buffer.
Discussed with: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25288
- Make use of cursors to avoid data copies for AES-CCM and AES-GCM.
Pass pointers into the request's input and/or output buffers
directly to the Update, encrypt, and decrypt hooks rather than
always copying all data into a temporary block buffer on the stack.
- Move handling for partial final blocks out of the main loop.
This removes branches from the main loop and permits using
encrypt/decrypt_last which avoids a memset to clear the rest of the
block on the stack.
- Shrink the on-stack buffers to assume AES block sizes and CCM/GCM
tag lengths.
- For AAD data, pass larger chunks to axf->Update. CCM can take each
AAD segment in a single call. GMAC can take multiple blocks at a
time.
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D25058
- crypto_apply() is only used for reading a buffer to compute a
digest, so change the data pointer to a const pointer.
- To better match m_apply(), change the data pointer type to void *
and the length from uint16_t to u_int. The length field in
particular matters as none of the apply logic was splitting requests
larger than UINT16_MAX.
- Adjust the auth_xform Update callback to match the function
prototype passed to crypto_apply() and crypto_apply_buf(). This
removes the needs for casts when using the Update callback.
- Change the Reinit and Setkey callbacks to also use a u_int length
instead of uint16_t.
- Update auth transforms for the changes. While here, use C99
initializers for auth_hash structures and avoid casts on callbacks.
Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25171
Some crypto consumers such as GELI and KTLS for file-backed sendfile
need to store their output in a separate buffer from the input.
Currently these consumers copy the contents of the input buffer into
the output buffer and queue an in-place crypto operation on the output
buffer. Using a separate output buffer avoids this copy.
- Create a new 'struct crypto_buffer' describing a crypto buffer
containing a type and type-specific fields. crp_ilen is gone,
instead buffers that use a flat kernel buffer have a cb_buf_len
field for their length. The length of other buffer types is
inferred from the backing store (e.g. uio_resid for a uio).
Requests now have two such structures: crp_buf for the input buffer,
and crp_obuf for the output buffer.
- Consumers now use helper functions (crypto_use_*,
e.g. crypto_use_mbuf()) to configure the input buffer. If an output
buffer is not configured, the request still modifies the input
buffer in-place. A consumer uses a second set of helper functions
(crypto_use_output_*) to configure an output buffer.
- Consumers must request support for separate output buffers when
creating a crypto session via the CSP_F_SEPARATE_OUTPUT flag and are
only permitted to queue a request with a separate output buffer on
sessions with this flag set. Existing drivers already reject
sessions with unknown flags, so this permits drivers to be modified
to support this extension without requiring all drivers to change.
- Several data-related functions now have matching versions that
operate on an explicit buffer (e.g. crypto_apply_buf,
crypto_contiguous_subsegment_buf, bus_dma_load_crp_buf).
- Most of the existing data-related functions operate on the input
buffer. However crypto_copyback always writes to the output buffer
if a request uses a separate output buffer.
- For the regions in input/output buffers, the following conventions
are followed:
- AAD and IV are always present in input only and their
fields are offsets into the input buffer.
- payload is always present in both buffers. If a request uses a
separate output buffer, it must set a new crp_payload_start_output
field to the offset of the payload in the output buffer.
- digest is in the input buffer for verify operations, and in the
output buffer for compute operations. crp_digest_start is relative
to the appropriate buffer.
- Add a crypto buffer cursor abstraction. This is a more general form
of some bits in the cryptosoft driver that tried to always use uio's.
However, compared to the original code, this avoids rewalking the uio
iovec array for requests with multiple vectors. It also avoids
allocate an iovec array for mbufs and populating it by instead walking
the mbuf chain directly.
- Update the cryptosoft(4) driver to support separate output buffers
making use of the cursor abstraction.
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24545
Add a 'native_blocksize' member to 'struct enc_xform' that ciphers can
use if they support a partial final block. This is particular useful
for stream ciphers, but can also apply to other ciphers. cryptosoft
will only pass in native blocks to the encrypt and decrypt hooks. For
the final partial block, 'struct enc_xform' now has new
encrypt_last/decrypt_last hooks which accept the length of the final
block. The multi_block methods are also retired.
Mark AES-ICM (AES-CTR) as a stream cipher. This has some interesting
effects on IPsec in that FreeBSD can now properly receive all packets
sent by Linux when using AES-CTR, but FreeBSD can no longer
interoperate with OpenBSD and older verisons of FreeBSD which assume
AES-CTR packets have a payload padded to a 16-byte boundary. Kornel
has offered to work on a patch to add a compatiblity sysctl to enforce
additional padding for AES-CTR in esp_output to permit compatibility
with OpenBSD and older versions of FreeBSD.
AES-XTS continues to use a block size of a single AES block length.
It is possible to adjust it to support partial final blocks by
implementing cipher text stealing via encrypt_last/decrypt_last hooks,
but I have not done so.
Reviewed by: cem (earlier version)
Tested by: Kornel Dulęba <mindal@semihalf.com> (AES-CTR with IPsec)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24906
- Consistently use 'void *' for key schedules / key contexts instead
of a mix of 'caddr_t', 'uint8_t *', and 'void *'.
- Add a ctxsize member to enc_xform similar to what auth transforms use
and require callers to malloc/zfree the context. The setkey callback
now supplies the caller-allocated context pointer and the zerokey
callback is removed. Callers now always use zfree() to ensure
key contexts are zeroed.
- Consistently use C99 initializers for all statically-initialized
instances of 'struct enc_xform'.
- Change the encrypt and decrypt functions to accept separate in and
out buffer pointers. Almost all of the backend crypto functions
already supported separate input and output buffers and this makes
it simpler to support separate buffers in OCF.
- Remove xform_userland.h shim to permit transforms to be compiled in
userland. Transforms no longer call malloc/free directly.
Reviewed by: cem (earlier version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24855
There are no in-kernel consumers.
Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24775
It no longer has any in-kernel consumers.
Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24772
They no longer have any in-tree consumers. Note that these are a
different from MD5-HMAC and SHA1-HMAC and were only used with IPsec.
Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24770
This was removed from IPsec in r286100 and no longer has any in-tree
consumers.
Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24769
