Right now, aesni_cipher_alloc does a bit of special-casing
for CRYPTO_F_IOV, to not do any allocation if the first uio
is large enough for the requested size. While working on ZFS
crypto port, I ran into horrible performance because the code
uses scatter-gather, and many of the times the data to encrypt
was in the second entry. This code looks through the list, and
tries to see if there is a single uio that can contain the
requested data, and, if so, uses that.
This has a slight impact on the current consumers, in that the
check is a little more complicated for the ones that use
CRYPTO_F_IOV -- but none of them meet the criteria for testing
more than one.
Submitted by: sef at ixsystems.com
Reviewed by: cem@
MFC after: 3 days
Sponsored by: iX Systems
Differential Revision: https://reviews.freebsd.org/D18522
As part of ZFS Crypto, I started getting a series of panics when I did not
have AESNI loaded. Adding locking fixed it, and I concluded that the
Reinit function altered the AES key schedule. This locking is not as
fine-grained as it could be (AESNI uses per-cpu locking), but
it's minimally invasive.
Sponsored by: iXsystems Inc
Reviewed by: cem, mav
Approved by: re (gjb), mav (mentor)
Differential Revision: https://reviews.freebsd.org/D17307
Fix a regression introduced in r336439.
Rather than allowing any linked list of algorithms, allow at most two
(typically, some combination of encrypt and/or MAC). Removes a WAITOK
malloc in an unsleepable context (classic LOR) by placing both software
algorithm contexts within the OCF-managed session object.
Tested with 'cryptocheck -a all -d cryptosoft0', which includes some
encrypt-and-MAC modes.
PR: 230304
Reported by: sef@
The wrapper is a thin shim around libsodium's Poly-1305 implementation. For
now, we just use the C algorithm and do not attempt to build the
SSE-optimized variant for x86 processors.
The algorithm support has not yet been plumbed through cryptodev, or added
to cryptosoft.
The timespecadd(3) family of macros were imported from NetBSD back in
r35029. However, they were initially guarded by #ifdef _KERNEL. In the
meantime, we have grown at least 28 syscalls that use timespecs in some
way, leading many programs both inside and outside of the base system to
redefine those macros. It's better just to make the definitions public.
Our kernel currently defines two-argument versions of timespecadd and
timespecsub. NetBSD, OpenBSD, and FreeDesktop.org's libbsd, however, define
three-argument versions. Solaris also defines a three-argument version, but
only in its kernel. This revision changes our definition to match the
common three-argument version.
Bump _FreeBSD_version due to the breaking KPI change.
Discussed with: cem, jilles, ian, bde
Differential Revision: https://reviews.freebsd.org/D14725
Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers. Avoid redundancy and complexity in
individual drivers by allocating session memory in the framework and
providing it to drivers in ::newsession().
Session handles are no longer integers with information encoded in various
high bits. Use of the CRYPTO_SESID2FOO() macros should be replaced with the
appropriate crypto_ses2foo() function on the opaque session handle.
Convert OCF drivers (in particular, cryptosoft, as well as myriad others) to
the opaque handle interface. Discard existing session tracking as much as
possible (quick pass). There may be additional code ripe for deletion.
Convert OCF consumers (ipsec, geom_eli, krb5, cryptodev) to handle-style
interface. The conversion is largely mechnical.
The change is documented in crypto.9.
Inspired by
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .
No objection from: ae (ipsec portion)
Reported by: jhb
In part, to support OpenSSL's use of cryptodev, which puts the HMAC pieces
in software and only offloads the raw hash primitive.
The following cryptodev identifiers are added:
* CRYPTO_RIPEMD160 (not hooked up)
* CRYPTO_SHA2_224
* CRYPTO_SHA2_256
* CRYPTO_SHA2_384
* CRYPTO_SHA2_512
The plain SHA1 and 2 hashes are plumbed through cryptodev (feels like there
is a lot of redundancy here...) and cryptosoft.
This adds new auth_hash implementations for the plain hashes, as well as
SHA1 (which had a cryptodev.h identifier, but no implementation).
Add plain SHA 1 and 2 hash tests to the cryptocheck tool.
Motivation stems from John Baldwin's earlier OCF email,
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .
When a caller passes in a uio or mbuf chain that is longer than crd_len, in
tandem with a transform that supports the multi-block interface,
swcr_encdec() would process the entire mbuf or uio instead of just the
portion indicated by crd_len (+ crd_skip).
De/encryption are performed in-place, so this would trash subsequent uio or
mbuf contents.
This was introduced in r331639 (mea culpa). It only affects the
{de,en}crypt_multi() family of interfaces. That interface only has one
consumer transform in-tree (for now): Chacha20.
PR: 227605
Submitted by: Valentin Vergez <valentin.vergez AT stormshield.eu>
opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
closer to "just about everywhere" than "only some files" per the
guidance in sys/conf/options.
Keep COMPAT_LINUX32 in opt_compat.h as it is confined to a subset of
sys/compat/linux/*.c. A fake _COMPAT_LINUX option ensure opt_compat.h
is created on all architectures.
Move COMPAT_LINUXKPI to opt_dontuse.h as it is only used to control the
set of compiled files.
Reviewed by: kib, cem, jhb, jtl
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14941
Introduced in r331639 by removing an instance of undefined behavior.
While we're here, the variable scope can be entirely moved inside the loop.
Reported by: Coverity
CID: 1387985
Sponsored by: Dell EMC Isilon
Mostly this is a thin shim around existing code to integrate with enc_xform
and cryptosoft (+ cryptodev).
Expand the cryptodev buffer used to match that of Chacha20's native block
size as a performance enhancement for chacha20_xform_crypt_multi.
xforms that support processing of multiple blocks at a time (to support more
efficient modes, for example) can define the encrypt_ and decrypt_multi
interfaces. If these interfaces are not present, the generic cryptosoft
code falls back on the block-at-a-time encrypt/decrypt interfaces.
Stream ciphers may support arbitrarily sized inputs (equivalent to an input
block size of 1 byte) but may be more efficient if a larger block is passed.
Sponsored by: Dell EMC Isilon
No functional change for Skipjack, AES-ICM, Blowfish, CAST-128, Camellia,
DES3, Rijndael128, DES. All of these have identical IV and blocksizes
declared in the associated enc_xform.
Functional changes for:
* AES-GCM: block len of 1, IV len of 12
* AES-XTS: block len of 16, IV len of 8
* NULL: block len of 4, IV len of 0
For these, it seems like the IV specified in the enc_xform is correct (and
the blocksize used before was wrong).
Additionally, the not-yet-OCFed cipher Chacha20 has a logical block length
of 1 byte, and a 16 byte IV + nonce.
Rationalize references to IV lengths to refer to the declared ivsize, rather
than declared blocksize.
Sponsored by: Dell EMC Isilon
The upstream repository is on github BLAKE2/libb2. Files landed in
sys/contrib/libb2 are the unmodified upstream files, except for one
difference: secure_zero_memory's contents have been replaced with
explicit_bzero() only because the previous implementation broke powerpc
link. Preferential use of explicit_bzero() is in progress upstream, so
it is anticipated we will be able to drop this diff in the future.
sys/crypto/blake2 contains the source files needed to port libb2 to our
build system, a wrapped (limited) variant of the algorithm to match the API
of our auth_transform softcrypto abstraction, incorporation into the Open
Crypto Framework (OCF) cryptosoft(4) driver, as well as an x86 SSE/AVX
accelerated OCF driver, blake2(4).
Optimized variants of blake2 are compiled for a number of x86 machines
(anything from SSE2 to AVX + XOP). On those machines, FPU context will need
to be explicitly saved before using blake2(4)-provided algorithms directly.
Use via cryptodev / OCF saves FPU state automatically, and use via the
auth_transform softcrypto abstraction does not use FPU.
The intent of the OCF driver is mostly to enable testing in userspace via
/dev/crypto. ATF tests are added with published KAT test vectors to
validate correctness.
Reviewed by: jhb, markj
Obtained from: github BLAKE2/libb2
Differential Revision: https://reviews.freebsd.org/D14662
An OCF-naive user program could use these primitives to implement HMAC, for
example. This would make the freed context sensitive data.
Probably other bzeros in this file should be explicit_bzeros as well.
Future work.
Reviewed by: jhb, markj
Differential Revision: https://reviews.freebsd.org/D14662 (minor part of a larger work)
Create a struct cryptop_data which contains state needed for a single
symmetric crypto operation and move that state out of the session. This
closes a race with the CRYPTO_F_DONE flag that can result in use after
free.
While here, remove the 'cse->error' member. It was just a copy of
'crp->crp_etype' and cryptodev_op() and cryptodev_aead() checked both
'crp->crp_etype' and 'cse->error'. Similarly, do not check for an
error from mtx_sleep() since it is not used with PCATCH or a timeout
so cannot fail with an error.
PR: 218597
Reviewed by: kib
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D13928
This adds explicit crp_mbuf and crp_uio pointers of the right type to
replace casts of crp_buf. This does not sweep through changing existing
code, but new code should use the correct fields instead of casts.
Reviewed by: kib
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D13927
Opaque pointers should be void *. Note that this does not go through
the tree removing all of the now-unnecessary casts.
Reviewed by: kib
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D13848
Just copyin the IV into the crypto descriptor directly. This avoids
copying the IV twice for each operation.
Reviewed by: kib
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D13847
In particular, no probes were present for AEAD requests, but also for
some other error cases in other ioctl requests.
MFC after: 2 weeks
Sponsored by: Chelsio Communications
This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.
Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385
fine when a lot of different flows to be ciphered/deciphered are involved.
However, when a software crypto driver is used, there are
situations where we could benefit from making crypto(9) multi threaded:
- a single flow is to be ciphered: only one thread is used to cipher it,
- a single ESP flow is to be deciphered: only one thread is used to
decipher it.
The idea here is to call crypto(9) using a new mode (CRYPTO_F_ASYNC) to
dispatch the crypto jobs on multiple threads, if the underlying crypto
driver is working in synchronous mode.
Another flag is added (CRYPTO_F_ASYNC_KEEPORDER) to make crypto(9)
dispatch the crypto jobs in the order they are received (an additional
queue/thread is used), so that the packets are reinjected in the network
using the same order they were posted.
A new sysctl net.inet.ipsec.async_crypto can be used to activate
this new behavior (disabled by default).
Submitted by: Emeric Poupon <emeric.poupon@stormshield.eu>
Reviewed by: ae, jmg, jhb
Differential Revision: https://reviews.freebsd.org/D10680
Sponsored by: Stormshield
A misordering in the Via padlock driver really strongly suggested that these
should use C99 named initializers.
No functional change.
Sponsored by: Dell EMC Isilon
Theoretically, HMACs do not actually have any limit on key sizes.
Transforms should compact input keys larger than the HMAC block size by
using the transform (hash) on the input key.
(Short input keys are padded out with zeros to the HMAC block size.)
Still, not all FreeBSD crypto drivers that provide HMAC functionality
handle longer-than-blocksize keys appropriately, so enforce a "maximum" key
length in the crypto API for auth_hashes that previously expressed a
requirement. (The "maximum" is the size of a single HMAC block for the
given transform.) Unconstrained auth_hashes are left as-is.
I believe the previous hardcoded sizes were committed in the original
import of opencrypto from OpenBSD and are due to specific protocol
details of IPSec. Note that none of the previous sizes actually matched
the appropriate HMAC block size.
The previous hardcoded sizes made the SHA tests in cryptotest.py
useless for testing FreeBSD crypto drivers; none of the NIST-KAT example
inputs had keys sized to the previous expectations.
The following drivers were audited to check that they handled keys up to
the block size of the HMAC safely:
Software HMAC:
* padlock(4)
* cesa
* glxsb
* safe(4)
* ubsec(4)
Hardware accelerated HMAC:
* ccr(4)
* hifn(4)
* sec(4) (Only supports up to 64 byte keys despite claiming to
support SHA2 HMACs, but validates input key sizes)
* cryptocteon (MIPS)
* nlmsec (MIPS)
* rmisec (MIPS) (Amusingly, does not appear to use key material at
all -- presumed broken)
Reviewed by: jhb (previous version), rlibby (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12437
When crypto_newsession() is given a request for an unsupported capability,
raise a more specific error than EINVAL.
This allows cryptotest.py to skip some HMAC tests that a driver does not
support.
Reviewed by: jhb, rlibby
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12451
In particular, support chaining an AES cipher with an HMAC for a request
including AAD. This permits submitting requests from userland to encrypt
objects like IPSec packets using these algorithms.
In the non-GCM case, the authentication crypto descriptor covers both the
AAD and the ciphertext. The GCM case remains unchanged. This matches
the requests created internally in IPSec. For the non-GCM case, the
COP_F_CIPHER_FIRST is also supported since the ordering matters.
Note that while this can be used to simulate IPSec requests from userland,
this ioctl cannot currently be used to perform TLS requests using AES-CBC
and MAC-before-encrypt.
Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11759
This requests that the cipher be performed before rather than after
the HMAC when both are specified for a single operation.
Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11757
Software crypto implementations don't care how the buffer is laid out,
but hardware implementations may assume that the AAD is always before
the plain/cipher text and that the hash/tag is immediately after the end
of the plain/cipher text.
In particular, this arrangement matches the layout of both IPSec packets
and TLS frames. Linux's crypto framework also assumes this layout for
AEAD requests.
Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11758
- Mark the source buffer for a copyback operation as const in the kernel
API.
- Use const with input-only buffers in crypto ioctl structures used with
/dev/crypto.
MFC after: 1 month
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D10517
The header was added by the recent keybuf feature (r316343)
MODINFOMD_KEYBUF originally resided here, but was moved to linker.h
This change fixes the build on risc-5 which doesn't have a metadata.h
Detected by Jenkins: https://ci.freebsd.org/job/FreeBSD-head-riscv64-build/1167/console
Reported by: lwhsu
This patch adds a general mechanism for providing encryption keys to the
kernel from the boot loader. This is intended to enable GELI support at
boot time, providing a better mechanism for passing keys to the kernel
than environment variables. It is designed to be extensible to other
applications, and can easily handle multiple encrypted volumes with
different keys.
This mechanism is currently used by the pending GELI EFI work.
Additionally, this mechanism can potentially be used to interface with
GRUB, opening up options for coreboot+GRUB configurations with completely
encrypted disks.
Another benefit over the existing system is that it does not require
re-deriving the user key from the password at each boot stage.
Most of this patch was written by Eric McCorkle. It was extended by
Allan Jude with a number of minor enhancements and extending the keybuf
feature into boot2.
GELI user keys are now derived once, in boot2, then passed to the loader,
which reuses the key, then passes it to the kernel, where the GELI module
destroys the keybuf after decrypting the volumes.
Submitted by: Eric McCorkle <eric@metricspace.net> (Original Version)
Reviewed by: oshogbo (earlier version), cem (earlier version)
MFC after: 3 weeks
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D9575
VFP code to store the old context, with lazy loading of the new context
when needed.
FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.
Reviewed by: kib
Obtained from: ABT Systeems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8276
This error looks like it was a simple copy-paste typo in the original commit
for this code (r275732).
PR: 204009
Reported by: Chang-Hsien Tsai <luke.tw AT gmail.com>
Sponsored by: EMC / Isilon Storage Division
Keep xform.c as a meta-file including the broken out bits
existing code that includes xform.c continues to work as normal
Individual algorithms can now be reused elsewhere, including outside
of the kernel
Reviewed by: bapt (previous version), gnn, delphij
Approved by: secteam
MFC after: 1 week
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D4674
cperciva's libmd implementation is 5-30% faster
The same was done for SHA256 previously in r263218
cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation
Extend sbin/md5 to create sha384(1)
Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}
Reviewed by: cperciva, des, delphij
Approved by: secteam, bapt (mentor)
MFC after: 2 weeks
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3929
Set zero ivsize for enc_xform_null and remove special handling from
xform_esp.c.
Reviewed by: gnn
Differential Revision: https://reviews.freebsd.org/D1503
