Commit Graph

304 Commits

Author SHA1 Message Date
Mark Johnston
c89def05b5 armv8crypto: Remove leftover debug printfs
Fixes:	26b08c5d21 ("armv8crypto: Use cursors to access crypto buffer data")
Reported by:	bz
2022-02-17 14:25:45 -05:00
Mark Johnston
26b08c5d21 armv8crypto: Use cursors to access crypto buffer data
Currently armv8crypto copies the scheme used in aesni(9), where payload
data and output buffers are allocated on the fly if the crypto buffer is
not virtually contiguous.  This scheme is simple but incurs a lot of
overhead: for an encryption request with a separate output buffer we
have to
- allocate a temporary buffer to hold the payload
- copy input data into the buffer
- copy the encrypted payload to the output buffer
- zero the temporary buffer before freeing it

We have a handy crypto buffer cursor abstraction now, so reimplement the
armv8crypto routines using that instead of temporary buffers.  This
introduces some extra complexity, but gallatin@ reports a 10% throughput
improvement with a KTLS workload without additional CPU usage.  The
driver still allocates an AAD buffer for AES-GCM if necessary.

Reviewed by:	jhb
Tested by:	gallatin
Sponsored by:	Ampere Computing LLC
Submitted by:	Klara Inc.
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D28950
2022-02-15 21:50:41 -05:00
Mark Johnston
0b3235ef74 armv8crypto: Factor out some duplicated GCM code
This is in preparation for using buffer cursors.  No functional change
intended.

Reviewed by:	jhb
Sponsored by:	Ampere Computing LLC
Submitted by:	Klara Inc.
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D28948
2022-02-15 21:47:41 -05:00
John Baldwin
0c6274a819 crypto: Add an API supporting curve25519.
This adds a wrapper around libsodium's curve25519 support matching
Linux's curve25519 API.  The intended use case for this is WireGuard.

Note that this is not integrated with OCF as it is not related to
symmetric operations on data.

Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33935
2022-01-24 15:27:39 -08:00
John Baldwin
89e0ee0db4 chacha20_poly1305: Use the correct license disclaimer.
Reviewed by:	emaste
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33976
2022-01-20 14:36:48 -08:00
John Baldwin
e71680049b crypto: Add a simple API for [X]ChaCha20-Poly1035 on flat buffers.
This is a synchronous software API which wraps the existing software
implementation shared with OCF.  Note that this will not currently
use optimized backends (such as ossl(4)) but may be appropriate for
operations on small buffers.

Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33524
2022-01-18 14:47:13 -08:00
John Baldwin
d7f0b3ce6d crypto: Re-add encrypt/decrypt_multi hooks to enc_xform.
These callbacks allow multiple contiguous blocks to be manipulated in
a single call.  Note that any trailing partial block for a stream
cipher must still be passed to encrypt/decrypt_last.

While here, document the setkey and reinit hooks and reorder the hooks
in 'struct enc_xform' to better reflect the life cycle.

Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33529
2022-01-11 14:17:41 -08:00
John Baldwin
e7d4d58030 sys/crypto: Use C99 fixed-width integer types.
No functional change.

Reviewed by:	imp
Differential Revision:	https://reviews.freebsd.org/D33633
2021-12-28 09:41:38 -08:00
Mark Johnston
4a61d8ef42 aesni: Fix an out-of-bounds read in AES_GCM_decrypt()
This is the same as 4285655adb ("aesni: Avoid a potential
out-of-bounds load in AES_GCM_encrypt()") except for the decryption
path.

Reported by:	Jenkins (KASAN job)
Reviewed by:	cem
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33474
2021-12-16 09:17:06 -05:00
Andrew Turner
71bf1c4cc5 Add accelerated arm64 sha512 to libmd
As with sha256 add support for accelerated sha512 support to libmd on
arm64. This depends on clang 13+ to build as this is the first release
with the needed intrinsics. Gcc should also support them, however from
a currently unknown release.

Reviewed by:	cem
Sponsored by:	The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33373
2021-12-13 15:33:22 +00:00
John Baldwin
6113a08b98 cryptosoft: Fully support per-operation keys for auth algorithms.
Only pre-allocate auth contexts when a session-wide key is provided or
for sessions without keys.  For sessions with per-operation keys,
always initialize the on-stack context directly rather than
initializing the session context in swcr_authprepare (now removed) and
then copying that session context into the on-stack context.

This approach permits parallel auth operations without needing a
serializing lock.  In addition, the previous code assumed that auth
sessions always provided an initial key unlike cipher sessions which
assume either an initial key or per-op keys.

While here, fix the Blake2 auth transforms to function like other auth
transforms where Setkey is invoked after Init rather than before.

Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33316
2021-12-09 11:52:42 -08:00
John Baldwin
6e17a2e00d crypto: Validate AES-GCM IV length in check_csp().
This centralizes the check for valid nonce lengths for AES-GCM.

While here, remove some duplicate checks for valid AES-GCM tag lengths
from ccp(4) and ccr(4).

Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33194
2021-12-09 11:52:41 -08:00
John Baldwin
a6765c4b44 libmd: Only define SHA256_Transform_c when using the ARM64 ifunc.
GCC 9 doesn't define a SHA256_Transform symbol when the stub just wraps
SHA256_Transform_c resulting in an undefined symbol for
_libmd_SHA256_Transform in libmd.so.

Discussed with:	andrew, jrtc27
Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D31945
2021-12-08 14:25:02 -08:00
Mateusz Guzik
3279341dc5 aesni: plug a set-but-not-unused var in aesni_newsession
Sponsored by:	Rubicon Communications, LLC ("Netgate")
2021-11-24 16:25:50 +00:00
Piotr Kubaj
3a60869237 Add assembly optimized code for OpenSSL on powerpc, powerpc64 and powerpc64le
Summary:
1. 34ab13b7d8
needs to be merged for ELFv2 support on big-endian.
2. crypto/openssl/crypto/ppccap.c needs to be patched.
Same reason as in https://github.com/openssl/openssl/pull/17082.

Approved by:	jkim, jhibbits
MFC after:	1 month
Differential Revision: https://reviews.freebsd.org/D33076
2021-11-23 23:26:53 +01:00
Mark Johnston
4285655adb aesni: Avoid a potential out-of-bounds load in AES_GCM_encrypt()
Reported by:	Jenkins (KASAN job)
Reviewed by:	cem, jhb
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33012
2021-11-16 13:30:22 -05:00
Kornel Duleba
c4026909bd ossl: Add support for ETA mode
Now that the AES-CBC is supported we can handle ETA requests.

Sponsored by:		Stormshield
Obtained from:		Semihalf
Reviewed by:		jhb
Differential revision:	https://reviews.freebsd.org/D32100
2021-11-08 10:53:36 +01:00
Kornel Duleba
197ff4c35b ossl: Add support for AES-CBC cipher
AES-CBC OpenSSL assembly is used underneath.
The glue layer(ossl_aes.c) is based on CHACHA20 implementation.
Contrary to the SHA and CHACHA20, AES OpenSSL assembly logic
does not have a fallback implementation in case CPU doesn't
support required instructions.
Because of that CPU caps are checked during initialization and AES
support is advertised only if available.
The feature is available on all architectures that ossl supports:
i386, amd64, arm64.

The biggest advantage of this patch over existing solutions
(aesni(4) and armv8crypto(4)) is that it supports SHA,
allowing for ETA operations.

Sponsored by:		Stormshield
Obtained from:		Semihalf
Reviewed by:		jhb (previous version)
Differential revision:	https://reviews.freebsd.org/D32099
2021-11-08 10:53:31 +01:00
Wojciech Macek
200bc58953 Revert "ossl: Add support for AES-CBC cipher"
This reverts commit 849faf4e0b.
2021-11-06 17:46:01 +01:00
Wojciech Macek
0dedcdaa1a Revert "ossl: Add support for ETA mode"
This reverts commit 048a71b46e.
2021-11-06 17:45:50 +01:00
Kornel Duleba
048a71b46e ossl: Add support for ETA mode
Now that the AES-CBC is supported we can handle ETA requests.

Sponsored by:		Stormshield
Obtained from:		Semihalf
Reviewed by:		jhb(previous version)
Differential revision:	https://reviews.freebsd.org/D32100
2021-11-06 09:08:44 +01:00
Kornel Duleba
849faf4e0b ossl: Add support for AES-CBC cipher
AES-CBC OpenSSL assembly is used underneath.
The glue layer(ossl_aes.c) is based on CHACHA20 implementation.
Contrary to the SHA and CHACHA20, AES OpenSSL assembly logic
does not have a fallback implementation in case CPU doesn't
support required instructions.
Because of that CPU caps are checked during initialization and AES
support is advertised only if available.
The feature is available on all architectures that ossl supports:
i386, amd64, arm64.

The biggest advantage of this patch over existing solutions
(aesni(4) and armv8crypto(4)) is that it supports SHA,
allowing for ETA operations.

Sponsored by:		Stormshield
Obtained from:		Semihalf
Reviewed by:		jhb
Differential revision:	https://reviews.freebsd.org/D32099
2021-11-06 09:08:44 +01:00
John Baldwin
42dcd39528 crypto: Support Chacha20-Poly1305 with a nonce size of 8 bytes.
This is useful for WireGuard which uses a nonce of 8 bytes rather
than the 12 bytes used for IPsec and TLS.

Note that this also fixes a (should be) harmless bug in ossl(4) where
the counter was incorrectly treated as a 64-bit counter instead of a
32-bit counter in terms of wrapping when using a 12 byte nonce.
However, this required a single message (TLS record) longer than 64 *
(2^32 - 1) bytes (about 256 GB) to trigger.

Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D32122
2021-10-06 14:08:49 -07:00
John Baldwin
655eb762c3 aesni: Support AES-CCM requests with a truncated tag.
Reviewed by:	sef
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D32115
2021-10-06 14:08:48 -07:00
John Baldwin
c09c379c7a aesni: Permit AES-CCM requests with neither payload nor AAD.
Reviewed by:	sef
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D32114
2021-10-06 14:08:47 -07:00
John Baldwin
d718c2d3c8 aesni: Handle requests with an empty payload.
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D32113
2021-10-06 14:08:47 -07:00
John Baldwin
8e6af6adfc aesni: Support multiple nonce lengths for AES-CCM.
Reviewed by:	sef
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D32112
2021-10-06 14:08:47 -07:00
John Baldwin
1833d6042c crypto: Permit variable-sized IVs for ciphers with a reinit hook.
Add a 'len' argument to the reinit hook in 'struct enc_xform' to
permit support for AEAD ciphers such as AES-CCM and Chacha20-Poly1305
which support different nonce lengths.

Reviewed by:	markj
Sponsored by:	Chelsio Communications, The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D32105
2021-10-06 14:08:46 -07:00
John Baldwin
cb128893b9 ccp, ccr: Simplify drivers to assume an AES-GCM IV length of 12.
While here, use crypto_read_iv() in a few more places in ccr(4) that I
missed previously.

Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D32104
2021-10-06 14:08:46 -07:00
Gordon Bergling
88a3af4da1 crypto(4): Fix a few typos in camellia.c
- s/valiables/variables/

Obtained from:	NetBSD
MFC after:	3 days
2021-09-04 13:02:11 +02:00
Mark Johnston
564b6aa7fc aesni: Avoid a potential out-of-bounds load in aes_encrypt_icm()
Given a partial block at the end of a payload, aes_encrypt_icm() would
perform a 16-byte load of the residual into a temporary variable.  This
is unsafe in principle since the full block may cross a page boundary.
Fix the problem by copying the residual into a stack buffer first.

Reported by:	syzbot+b7e44cde9e2e89f0f6c9@syzkaller.appspotmail.com
Reported by:	syzbot+4b5eaf123a99456b5160@syzkaller.appspotmail.com
Reported by:	syzbot+70c74c1aa232633355ca@syzkaller.appspotmail.com
Reported by:	syzbot+2c663776a52828373d41@syzkaller.appspotmail.com
Reviewed by:	cem, jhb
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D31730
2021-08-30 14:22:20 -04:00
Andrew Turner
69d8dc20be Use arm64 sha256 intrinsics in libmd
Summary:
When running on a CPU that supports the arm64 sha256 intrinsics use them
to improve perfromance of sha256 calculations.

With this changethe following improvement has been seen on an Apple M1
with FreeBS running under Parallels, with similar results on a
Neoverse-N1 r3p1.

x sha256.orig
+ sha256.arm64
+--------------------------------------------------------------------+
|++                                                               x x|
|+++                                                              xxx|
||A                                                               |A||
+--------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5          3.41           3.5          3.46         3.458   0.042661458
+   5          0.47          0.54           0.5         0.504   0.027018512
Difference at 95.0% confidence
        -2.954 +/- 0.0520768
        -85.4251% +/- 0.826831%
        (Student's t, pooled s = 0.0357071)

Reviewed by:	cem
Sponsored by:	The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31284
2021-08-11 10:20:48 +00:00
Ed Maste
9feff969a0 Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights
These ones were unambiguous cases where the Foundation was the only
listed copyright holder (in the associated license block).

Sponsored by:	The FreeBSD Foundation
2021-08-08 10:42:24 -04:00
Mark Johnston
d8787d4f78 crypto: Constify all transform descriptors
No functional change intended.

Reviewed by:	ae, jhb
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D31196
2021-07-26 16:41:05 -04:00
John Baldwin
1c09320d58 ossl: Use crypto_cursor_segment().
Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30447
2021-05-25 16:59:19 -07:00
John Baldwin
883a0196b6 crypto: Add a new type of crypto buffer for a single mbuf.
This is intended for use in KTLS transmit where each TLS record is
described by a single mbuf that is itself queued in the socket buffer.
Using the existing CRYPTO_BUF_MBUF would result in
bus_dmamap_load_crp() walking additional mbufs in the socket buffer
that are not relevant, but generating a S/G list that potentially
exceeds the limit of the tag (while also wasting CPU cycles).

Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D30136
2021-05-25 16:59:18 -07:00
John Baldwin
d2e076c37b ossl: Don't encryt/decrypt too much data for chacha20.
The loops for Chacha20 and Chacha20+Poly1305 which encrypted/decrypted
full blocks of data used the minimum of the input and output segment
lengths to determine the size of the next chunk ('todo') to pass to
Chacha20_ctr32().  However, the input and output segments could extend
past the end of the ciphertext region into the tag (e.g.  if a "plain"
single mbuf contained an entire TLS record).  If the length of the tag
plus the length of the last partial block together were at least as
large as a full Chacha20 block (64 bytes), then an extra block was
encrypted/decrypted overlapping with the tag.  Fix this by also
capping the amount of data to encrypt/decrypt by the amount of
remaining data in the ciphertext region ('resid').

Reported by:	gallatin
Reviewed by:	cem, gallatin, markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D29517
2021-04-01 15:49:07 -07:00
Mitchell Horne
da45b46266 armv8crypto: note derivation in armv8_crypto_wrap.c
This file inherits some boilerplate and structure from the analogous
file in aesni(4), aesni_wrap.c. Note the derivation and the copyright
holders of that file.

For example, the AES-XTS bits added in 4979620ece were ported from
aesni(4).

Requested by:	jmg
Reviewed by:	imp, gnn
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D29268
2021-03-19 10:53:49 -03:00
Oleksandr Tymoshenko
748be78e60 armv8crypto: fix AES-XTS regression introduced by ed9b7f44
Initialization of the XTS key schedule was accidentally dropped
when adding AES-GCM support so all-zero schedule was used instead.
This rendered previously created GELI partitions unusable.
This change restores proper XTS key schedule initialization.

Reported by:	Peter Jeremy <peter@rulingia.com>
MFC after:	immediately
2021-03-07 12:03:47 -08:00
John Baldwin
78991a93eb ossl: Add support for the ChaCha20 + Poly1305 AEAD cipher from RFC 8439
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D28757
2021-03-03 15:20:57 -08:00
John Baldwin
92aecd1e6f ossl: Add ChaCha20 cipher support.
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D28756
2021-03-03 15:20:57 -08:00
John Baldwin
a079e38b08 ossl: Add Poly1305 digest support.
Reviewed by:	cem
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D28754
2021-03-03 15:20:57 -08:00
Elliott Mitchell
530d38441d armv8crypto: add missing newline
The missing newline mildly garbles boot-time messages and this can be
troublesome if you need those.

Fixes:		a520f5ca58 ("armv8crypto: print a message on probe failure")
Reported by:	Mike Karels (mike@karels.net)
Reviewed By:	gonzo
Differential Revision:	https://reviews.freebsd.org/D28988
2021-02-28 16:03:55 -04:00
Jung-uk Kim
3396647c8d OpenSSL: Regen assembly files for OpenSSL 1.1.1j 2021-02-17 00:35:11 -05:00
Mark Johnston
7509b677b4 armv8crypto: Extract GCM state into a structure
This makes it easier to refactor the GCM code to operate on
crypto_buffer_cursors rather than plain contiguous buffers, with the aim
of minimizing the amount of copying and zeroing done today.

No functional change intended.

Reviewed by:	jhb
MFC after:	1 week
Sponsored by:	Ampere Computing
Submitted by:	Klara, Inc.
Differential Revision:	https://reviews.freebsd.org/D28500
2021-02-08 09:19:10 -05:00
Mark Johnston
0dc7076037 armv8crypto: Fix some edge cases in the AES-GCM implementation
- We were only hashing up to the first 16 bytes of the AAD.
- When computing the digest during decryption, handle the case where
  len == trailer, i.e., len < AES_BLOCK_LEN, properly.

While here:

- trailer is always smaller than AES_BLOCK_LEN, so remove a pair of
  unnecessary modulus operations.
- Replace some byte-by-byte loops with memcpy() and memset() calls.
  In particular, zero the full block before copying a partial block into
  it since we do that elsewhere and it means that the memset() length is
  known at compile time.

Reviewed by:	jhb
Sponsored by:	Ampere Computing
Submitted by:	Klara, Inc.
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D28501
2021-02-08 09:19:07 -05:00
Oleksandr Tymoshenko
ed9b7f4414 armv8crypto: add AES-GCM support
Add support for AES-GCM using OpenSSL's accelerated routines.

Reviewed by:	jhb
Differential Revision:	https://reviews.freebsd.org/D27454
Sponsored by:	Ampere Computing
Submitted by:	Klara, Inc.
2021-01-21 21:41:27 -08:00
Mark Johnston
098c902b52 aesni: Ensure that key schedules are aligned
Rather than depending on malloc() returning 16-byte aligned chunks,
allocate some extra pad bytes and ensure that key schedules are
appropriately aligned.

Reviewed by:	kib
MFC after:	2 weeks
Sponsored by:	Rubicon Communications, LLC (Netgate)
Differential Revision:	https://reviews.freebsd.org/D28157
2021-01-18 17:07:56 -05:00
Mitchell Horne
a520f5ca58 armv8crypto: print a message on probe failure
Similar to the message printed by aesni(4), let the user know if the
driver is unsupported by their CPU.

PR:		252543
Reported by:	gbe
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2021-01-18 16:59:21 -04:00
Mitchell Horne
4979620ece armv8crypto: add AES-XTS support
A straightforward(ish) port from aesni(4). This implementation does not
perform loop unrolling on the input blocks, so this is left as a future
performance improvement.

Submitted by:	Greg V <greg AT unrelenting.technology>
Looks good:	jhb, jmg
Tested by:	mhorne
Differential Revision:	https://reviews.freebsd.org/D21017
2021-01-07 15:35:20 -04:00