The armv8crypto module includes arm_neon.h for the compiler intrinsic
functions. This includes the userland stdint.h file that doesn't exist in
the kernel. Fix this by providing an empty stdint.h to be used when we
include arm_neon.h.
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D16254
aesni(4) allocates a contiguous buffer for the data it processes if the
provided input was not already virtually contiguous, and copies the input
there. It performs encryption or decryption in-place.
r324037 removed the logic that then copied the processed data back to the
user-provided input buffer, breaking {de,enc}crypt for mbuf chains or
iovecs with more than a single descriptor.
PR: 228094 (probably, not confirmed)
Submitted by: Sean Fagan <kithrup AT me.com>
Reported by: Emeric POUPON <emeric.poupon AT stormshield.eu>
X-MFC-With: 324037
Security: could result in plaintext being output by "encrypt"
operation
Mostly this is a thin shim around existing code to integrate with enc_xform
and cryptosoft (+ cryptodev).
Expand the cryptodev buffer used to match that of Chacha20's native block
size as a performance enhancement for chacha20_xform_crypt_multi.
Drop our local patch and restore full vanilla upstream code in
contrib/libb2.
No functional change intended. explicit_bzero() should continue to be used.
Obtained from: libb2 b4b241a34824b51956a7866606329a065d397525
Sponsored by: Dell EMC Isilon
after the array to its proper location. Otherwise, the linker.hints
file has things out of order and we associated it with whatever was
the previous module.
The upstream repository is on github BLAKE2/libb2. Files landed in
sys/contrib/libb2 are the unmodified upstream files, except for one
difference: secure_zero_memory's contents have been replaced with
explicit_bzero() only because the previous implementation broke powerpc
link. Preferential use of explicit_bzero() is in progress upstream, so
it is anticipated we will be able to drop this diff in the future.
sys/crypto/blake2 contains the source files needed to port libb2 to our
build system, a wrapped (limited) variant of the algorithm to match the API
of our auth_transform softcrypto abstraction, incorporation into the Open
Crypto Framework (OCF) cryptosoft(4) driver, as well as an x86 SSE/AVX
accelerated OCF driver, blake2(4).
Optimized variants of blake2 are compiled for a number of x86 machines
(anything from SSE2 to AVX + XOP). On those machines, FPU context will need
to be explicitly saved before using blake2(4)-provided algorithms directly.
Use via cryptodev / OCF saves FPU state automatically, and use via the
auth_transform softcrypto abstraction does not use FPU.
The intent of the OCF driver is mostly to enable testing in userspace via
/dev/crypto. ATF tests are added with published KAT test vectors to
validate correctness.
Reviewed by: jhb, markj
Obtained from: github BLAKE2/libb2
Differential Revision: https://reviews.freebsd.org/D14662
Two copies of chacha20 were imported into the tree on Apr 15 2017 (r316982)
and Apr 16 2017 (r317015). Only the latter is actually used by anything, so
just go ahead and garbage collect the unused version while it's still only
in CURRENT.
I'm not making any judgement on which implementation is better. If I pulled
the wrong one, feel free to swap the existing implementation out and replace
it with the other code (conforming to the API that actually gets used in
randomdev, of course). We only need one generic implementation.
Sponsored by: Dell EMC Isilon
Improve clarity of a comment and style(9) some areas.
No functional change.
Reported by: markj (on review of a mostly-copied driver)
Sponsored by: Dell EMC Isilon
No implementation of fpu_kern_enter() can fail, and it was causing needless
error checking boilerplate and confusion. Change the return code to void to
match reality.
(This trivial change took nine days to land because of the commit hook on
sys/dev/random. Please consider removing the hook or otherwise lowering the
bar -- secteam never seems to have free time to review patches.)
Reported by: Lachlan McIlroy <Lachlan.McIlroy AT isilon.com>
Reviewed by: delphij
Approved by: secteam (delphij)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D14380
Apply r328361 to duplicate copy of ccr_gcm_soft in ccp(4).
Properly honor the lack of the CRD_F_IV_PRESENT flag in the GCM software
fallback case for encryption requests.
* Registers TRNG source for random(4)
* Finds available queues, LSBs; allocates static objects
* Allocates a shared MSI-X for all queues. The hardware does not have
separate interrupts per queue. Working interrupt mode driver.
* Computes SHA hashes, HMAC. Passes cryptotest.py, cryptocheck tests.
* Does AES-CBC, CTR mode, and XTS. cryptotest.py and cryptocheck pass.
* Support for "authenc" (AES + HMAC). (SHA1 seems to result in
"unaligned" cleartext inputs from cryptocheck -- which the engine
cannot handle. SHA2 seems to work fine.)
* GCM passes for block-multiple AAD, input lengths
Largely based on ccr(4), part of cxgbe(4).
Rough performance averages on AMD Ryzen 1950X (4kB buffer):
aesni: SHA1: ~8300 Mb/s SHA256: ~8000 Mb/s
ccp: ~630 Mb/s SHA256: ~660 Mb/s SHA512: ~700 Mb/s
cryptosoft: ~1800 Mb/s SHA256: ~1800 Mb/s SHA512: ~2700 Mb/s
As you can see, performance is poor in comparison to aesni(4) and even
cryptosoft (due to high setup cost). At a larger buffer size (128kB),
throughput is a little better (but still worse than aesni(4)):
aesni: SHA1:~10400 Mb/s SHA256: ~9950 Mb/s
ccp: ~2200 Mb/s SHA256: ~2600 Mb/s SHA512: ~3800 Mb/s
cryptosoft: ~1750 Mb/s SHA256: ~1800 Mb/s SHA512: ~2700 Mb/s
AES performance has a similar story:
aesni: 4kB: ~11250 Mb/s 128kB: ~11250 Mb/s
ccp: ~350 Mb/s 128kB: ~4600 Mb/s
cryptosoft: ~1750 Mb/s 128kB: ~1700 Mb/s
This driver is EXPERIMENTAL. You should verify cryptographic results on
typical and corner case inputs from your application against a known- good
implementation.
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12723
Always initialize some variables GCC warns about. They are initialized in
every path where they are used, but GCC doesn't know that.
Sponsored by: Dell EMC Isilon
directives (as reported in https://bugs.llvm.org/show_bug.cgi?id=35749),
by defining the rep prefix with yet another .byte directive.
This is a temporary fix, to be reverted before merging back to head,
until upstream has a proper fix for this.
Mainly focus on files that use BSD 3-Clause license.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
Remove some dead conditionals and add an assertion around behavior already
present in aesni_process().
Silence a few Coverity false positives.
CIDs: 1381571, 1381557
Sponsored by: Dell EMC Isilon
The GCC xmmintrin.h header brokenly includes mm_malloc.h unconditionally.
(The Clang version of xmmintrin.h only includes mm_malloc.h if not compiling
in standalone mode.)
Hack around GCC's broken header by defining the include guard macro ahead of
including xmmintrin.h.
Reported by: lwhsu, jhb
Tested by: lwhsu
Sponsored by: Dell EMC Isilon
Some x86 class CPUs have accelerated intrinsics for SHA1 and SHA256.
Provide this functionality on CPUs that support it.
This implements CRYPTO_SHA1, CRYPTO_SHA1_HMAC, and CRYPTO_SHA2_256_HMAC.
Correctness: The cryptotest.py suite in tests/sys/opencrypto has been
enhanced to verify SHA1 and SHA256 HMAC using standard NIST test vectors.
The test passes on this driver. Additionally, jhb's cryptocheck tool has
been used to compare various random inputs against OpenSSL. This test also
passes.
Rough performance averages on AMD Ryzen 1950X (4kB buffer):
aesni: SHA1: ~8300 Mb/s SHA256: ~8000 Mb/s
cryptosoft: ~1800 Mb/s SHA256: ~1800 Mb/s
So ~4.4-4.6x speedup depending on algorithm choice. This is consistent with
the results the Linux folks saw for 4kB buffers.
The driver borrows SHA update code from sys/crypto sha1 and sha256. The
intrinsic step function comes from Intel under a 3-clause BSDL.[0] The
intel_sha_extensions_sha<foo>_intrinsic.c files were renamed and lightly
modified (added const, resolved a warning or two; included the sha_sse
header to declare the functions).
[0]: https://software.intel.com/en-us/articles/intel-sha-extensions-implementations
Reviewed by: jhb
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12452
A misordering in the Via padlock driver really strongly suggested that these
should use C99 named initializers.
No functional change.
Sponsored by: Dell EMC Isilon
Theoretically, HMACs do not actually have any limit on key sizes.
Transforms should compact input keys larger than the HMAC block size by
using the transform (hash) on the input key.
(Short input keys are padded out with zeros to the HMAC block size.)
Still, not all FreeBSD crypto drivers that provide HMAC functionality
handle longer-than-blocksize keys appropriately, so enforce a "maximum" key
length in the crypto API for auth_hashes that previously expressed a
requirement. (The "maximum" is the size of a single HMAC block for the
given transform.) Unconstrained auth_hashes are left as-is.
I believe the previous hardcoded sizes were committed in the original
import of opencrypto from OpenBSD and are due to specific protocol
details of IPSec. Note that none of the previous sizes actually matched
the appropriate HMAC block size.
The previous hardcoded sizes made the SHA tests in cryptotest.py
useless for testing FreeBSD crypto drivers; none of the NIST-KAT example
inputs had keys sized to the previous expectations.
The following drivers were audited to check that they handled keys up to
the block size of the HMAC safely:
Software HMAC:
* padlock(4)
* cesa
* glxsb
* safe(4)
* ubsec(4)
Hardware accelerated HMAC:
* ccr(4)
* hifn(4)
* sec(4) (Only supports up to 64 byte keys despite claiming to
support SHA2 HMACs, but validates input key sizes)
* cryptocteon (MIPS)
* nlmsec (MIPS)
* rmisec (MIPS) (Amusingly, does not appear to use key material at
all -- presumed broken)
Reviewed by: jhb (previous version), rlibby (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12437
A long long time ago the register keyword told the compiler to store
the corresponding variable in a CPU register, but it is not relevant
for any compiler used in the FreeBSD world today.
ANSIfy related prototypes while here.
Reviewed by: cem, jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10193
In my eagerness to eliminate a branch which is taken once per 2^38
bytes of keystream, I forgot that the state words are in host order.
Thus, the counter increment code worked fine on little-endian
machines, but not on big-endian ones. Switch to a simpler (branchful)
solution.
numbers with Chacha20. Keep the API, though, as that is what the
other *BSD's have done.
Use the boot-time entropy stash (if present) to bootstrap the
in-kernel entropy source.
Reviewed by: delphij,rwatson
Approved by: so(delphij)
MFC after: 2 months
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D10048
This patch adds a general mechanism for providing encryption keys to the
kernel from the boot loader. This is intended to enable GELI support at
boot time, providing a better mechanism for passing keys to the kernel
than environment variables. It is designed to be extensible to other
applications, and can easily handle multiple encrypted volumes with
different keys.
This mechanism is currently used by the pending GELI EFI work.
Additionally, this mechanism can potentially be used to interface with
GRUB, opening up options for coreboot+GRUB configurations with completely
encrypted disks.
Another benefit over the existing system is that it does not require
re-deriving the user key from the password at each boot stage.
Most of this patch was written by Eric McCorkle. It was extended by
Allan Jude with a number of minor enhancements and extending the keybuf
feature into boot2.
GELI user keys are now derived once, in boot2, then passed to the loader,
which reuses the key, then passes it to the kernel, where the GELI module
destroys the keybuf after decrypting the volumes.
Submitted by: Eric McCorkle <eric@metricspace.net> (Original Version)
Reviewed by: oshogbo (earlier version), cem (earlier version)
MFC after: 3 weeks
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D9575
on the AES-NI code, and modified as needed for use on ARMv8. When loaded
the driver will check the appropriate field in the id_aa64isar0_el1
register to see if AES is supported, and if so the probe function will
signal the driver should attach.
With this I have seen up to 2000Mb/s from the cryptotest test with a single
thread on a ThunderX Pass 2.0.
Reviewed by: imp
Obtained from: ABT Systems Ltd
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8297
C99 allows array function parameters to use the static keyword for their
sizes. This tells the compiler that the parameter will have at least the
specified size, and calling code will fail to compile if that guarantee is
not met. However, this syntax is not legal in C++.
This commit reverts r300824, which worked around the problem for
sys/sys/md5.h only, and introduces a new macro: min_size(). min_size(x) can
be used in headers as a static array size, but will still compile in C++
mode.
Reviewed by: cem, ed
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D8277
Support for the new hashing algorithms in ZFS was introduced in r289422
However it was disconnected because FreeBSD lacked implementations of
SHA-512 (truncated to 256 bits), and Skein.
These implementations were introduced in r300921 and r300966 respectively
This commit connects them to ZFS and enabled these new checksum algorithms
This new algorithms are not supported by the boot blocks, so do not use them
on your root dataset if you boot from ZFS.
Relnotes: yes
Sponsored by: ScaleEngine Inc.
a preference for memory load instructions over large code footprints
with embedded immediate variables.
On amd64 CPUs from 2007-2008 there is not a significant change, but
amd64 CPUs from 2009-2010 get roughly 10% more throughput with this
code; amd64 CPUs from 2011-2012 get roughly 15% more throughput; and
AMD64 CPUs from 2013-2015 get 20-25% more throughput. The Raspberry
Pi 2 increases its throughput by 6-8%.
Sponsored by: Tarsnap Backup Inc.
Performance tested by: allanjude
MFC after: 3 weeks
Connect it to userland (libmd, libcrypt, sbin/md5) and kernel (crypto.ko)
Support for skein as a ZFS checksum algorithm was introduced in r289422
but is disconnected because FreeBSD lacked a Skein implementation.
A further commit will enable it in ZFS.
Reviewed by: cem
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D6166
This implements SHA-512/256, which generates a 256 bit hash by
calculating the SHA-512 then truncating the result. A different initial
value is used, making the result different from the first 256 bits of
the SHA-512 of the same input. SHA-512 is ~50% faster than SHA-256 on
64bit platforms, so the result is a faster 256 bit hash.
The main goal of this implementation is to enable support for this
faster hashing algorithm in ZFS. The feature was introduced into ZFS
in r289422, but is disconnected because SHA-512/256 support was missing.
A further commit will enable it in ZFS.
This is the follow on to r292782
Reviewed by: cem
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D6061
Use the C99 'static' keyword to hint to the compiler IVs and output digest
sizes. The keyword informs the compiler of the minimum valid size for a given
array. Obviously not every pointer can be validated (i.e., the compiler can
produce false negative but not false positive reports).
No functional change. No ABI change.
Sponsored by: EMC / Isilon Storage Division
Keep xform.c as a meta-file including the broken out bits
existing code that includes xform.c continues to work as normal
Individual algorithms can now be reused elsewhere, including outside
of the kernel
Reviewed by: bapt (previous version), gnn, delphij
Approved by: secteam
MFC after: 1 week
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D4674
cperciva's libmd implementation is 5-30% faster
The same was done for SHA256 previously in r263218
cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation
Extend sbin/md5 to create sha384(1)
Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}
Reviewed by: cperciva, des, delphij
Approved by: secteam, bapt (mentor)
MFC after: 2 weeks
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3929
