Commit Graph

208 Commits

Author SHA1 Message Date
Andrew Turner
88c66b953a Fix the armv8 crypto driver after r354170.
Sponsored by:	DARPA, AFRL
2019-10-30 10:41:10 +00:00
John Baldwin
87210a0c6d Stop using des_cblock * for arguments to DES functions.
This amounts to a char ** since it is a char[8] *.  Evil casts mostly
resolved the fact that what was actually passed in were plain char *.
Instead, change the DES functions to use 'unsigned char *' for keys
and for input and output buffers.

Reviewed by:	cem, imp
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21389
2019-08-26 17:25:07 +00:00
John Baldwin
821fe3d3a4 Use 'const' for keys and IVs passed to software encryption algorithms.
Specifically, use 'const' for the key passed to the 'setkey' method
and 'const' for the 'iv' passed to the 'reinit' method.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21347
2019-08-22 00:02:08 +00:00
Ed Maste
8279d4b711 armv8crypto: add missing unlock for session (after r336439)
Attempts to use cryptodev (e.g. tests at /usr/src/tests/sys/opencrypto
with armv8crypto added to the module lists) were causing a panic.

Submitted by:	Greg V <greg_unrelenting.technology>
Differential Revision:	https://reviews.freebsd.org/D21012
2019-08-08 14:34:53 +00:00
Conrad Meyer
6fe286ed83 aesni(4): Fix trivial type typo
This fixes the kernel build with xtoolchain-gcc (6.4.0).

X-MFC-With:	r348268
2019-05-27 00:47:51 +00:00
Sean Eric Fagan
7cff9f3748 Add an AESNI-optimized version of the CCM/CBC cryptographic and authentication
code.  The primary client of this is probably going to be ZFS encryption.

Reviewed by:	jhb, cem
Sponsored by:	iXsystems Inc, Kithrup Enterprises
Differential Revision:	https://reviews.freebsd.org/D19298
2019-05-25 07:26:30 +00:00
Conrad Meyer
e2e050c8ef Extract eventfilter declarations to sys/_eventfilter.h
This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.

EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).

As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions.  The remainder of the patch addresses
adding appropriate includes to fix those files.

LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).

No functional change (intended).  Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed.  __FreeBSD_version has been bumped.
2019-05-20 00:38:23 +00:00
Conrad Meyer
7d93ab5e35 Embedded chacha: Add 0-bit iv + 128-bit counter mode
This mode might be suitable for a Fortuna keystream primitive.

Reviewed by:	markm
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D19410
2019-03-01 23:30:23 +00:00
Warner Losh
52467047aa Regularize the Netflix copyright
Use recent best practices for Copyright form at the top of
the license:
1. Remove all the All Rights Reserved clauses on our stuff. Where we
   piggybacked others, use a separate line to make things clear.
2. Use "Netflix, Inc." everywhere.
3. Use a single line for the copyright for grep friendliness.
4. Use date ranges in all places for our stuff.

Approved by: Netflix Legal (who gave me the form), adrian@ (pmc files)
2019-02-04 21:28:25 +00:00
Matt Macy
ff2038a9bf Generalize AES iov optimization
Right now, aesni_cipher_alloc does a bit of special-casing
for CRYPTO_F_IOV, to not do any allocation if the first uio
is large enough for the requested size. While working on ZFS
crypto port, I ran into horrible performance because the code
uses scatter-gather, and many of the times the data to encrypt
was in the second entry. This code looks through the list, and
tries to see if there is a single uio that can contain the
requested data, and, if so, uses that.

This has a slight impact on the current consumers, in that the
check is a little more complicated for the ones that use
CRYPTO_F_IOV -- but none of them meet the criteria for testing
more than one.

Submitted by:	sef at ixsystems.com
Reviewed by:	cem@
MFC after:	3 days
Sponsored by:	iX Systems
Differential Revision:	https://reviews.freebsd.org/D18522
2018-12-13 04:40:53 +00:00
Conrad Meyer
070249043e rijndael (AES): Avoid leaking sensitive data on kernel stack
Noticed this investigating Fortuna.  Remove useless duplicate stack copies
of sensitive contents when possible, or if not possible, be sure to zero
them out when we're finished.

Approved by:	secteam (gordon)
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D16935
2018-10-26 20:53:01 +00:00
Conrad Meyer
f449384c4a Add explicit copyright text to trivial header
Reported by:	rgrimes
2018-10-22 01:27:11 +00:00
Conrad Meyer
987733eddf Embedded chacha: Remove some harmless dead stores in keystream mode
(From r338059.)

Sponsored by:	Dell EMC Isilon
2018-10-20 22:17:16 +00:00
Conrad Meyer
243e09430a Embedded chacha: Distinguish via dedicated macro
Set embedding expectations via CHACHA_EMBED macro rather than _KERNEL
definition.

No functional change.

Sponsored by:	Dell EMC Isilon
2018-10-20 22:14:29 +00:00
Conrad Meyer
addfc6364f crypto/chacha: Split header into separate _chacha.h
Sponsored by:	Dell EMC Isilon
2018-10-20 22:12:53 +00:00
Warner Losh
329e817fcc Reapply, with minor tweaks, r338025, from the original commit:
Remove unused and easy to misuse PNP macro parameter

Inspired by r338025, just remove the element size parameter to the
MODULE_PNP_INFO macro entirely.  The 'table' parameter is now required to
have correct pointer (or array) type.  Since all invocations of the macro
already had this property and the emitted PNP data continues to include the
element size, there is no functional change.

Mostly done with the coccinelle 'spatch' tool:

  $ cat modpnpsize0.cocci
    @normaltables@
    identifier b,c;
    expression a,d,e;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,d,
    -sizeof(d[0]),
     e);

    @singletons@
    identifier b,c,d;
    expression a;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,&d,
    -sizeof(d),
     1);

  $ rg -l MODULE_PNP_INFO -- sys | \
    xargs spatch --in-place --sp-file modpnpsize0.cocci

(Note that coccinelle invokes diff(1) via a PATH search and expects diff to
tolerate the -B flag, which BSD diff does not.  So I had to link gdiff into
PATH as diff to use spatch.)

Tinderbox'd (-DMAKE_JUST_KERNELS).
Approved by: re (glen)
2018-09-26 17:12:14 +00:00
Warner Losh
91eeadc516 Don't load ccp automatically with devmatch
Remove the PNP info for the moment from the driver. It's an
experimental driver (as noted in r328150). It's performance is about
1/10th that of aesni. It will often panic when used with GELI (PR
2279820).  It's not in our best interest to have such a driver be
autoloaded by default.

Approved by: re@ (rgrimes)
Reviewed By: cem@
Differential Review: https://reviews.freebsd.org/D16959
2018-08-31 01:01:16 +00:00
Xin LI
c1e80940f3 Update userland arc4random() with OpenBSD's Chacha20 based arc4random().
ObsoleteFiles.inc:

    Remove manual pages for arc4random_addrandom(3) and
    arc4random_stir(3).

  contrib/ntp/lib/isc/random.c:
  contrib/ntp/sntp/libevent/evutil_rand.c:

    Eliminate in-tree usage of arc4random_addrandom().

  crypto/heimdal/lib/roken/rand.c:
  crypto/openssh/config.h:

    Eliminate in-tree usage of arc4random_stir().

  include/stdlib.h:

    Remove arc4random_stir() and arc4random_addrandom() prototypes,
    provide temporary shims for transistion period.

  lib/libc/gen/Makefile.inc:

    Hook arc4random-compat.c to build, add hint for Chacha20 source for
    kernel, and remove arc4random_addrandom(3) and arc4random_stir(3)
    links.

  lib/libc/gen/arc4random.c:

    Adopt OpenBSD arc4random.c,v 1.54 with bare minimum changes, use the
    sys/crypto/chacha20 implementation of keystream.

  lib/libc/gen/Symbol.map:

    Remove arc4random_stir and arc4random_addrandom interfaces.

  lib/libc/gen/arc4random.h:

    Adopt OpenBSD arc4random.h,v 1.4 but provide _ARC4_LOCK of our own.

  lib/libc/gen/arc4random.3:

    Adopt OpenBSD arc4random.3,v 1.35 but keep FreeBSD r114444 and
    r118247.

  lib/libc/gen/arc4random-compat.c:

    Compatibility shims for arc4random_stir and arc4random_addrandom
    functions to preserve ABI.  Log once when called but do nothing
    otherwise.

  lib/libc/gen/getentropy.c:
  lib/libc/include/libc_private.h:

    Fold __arc4_sysctl into getentropy.c (renamed to arnd_sysctl).
    Remove from libc_private.h as a result.

  sys/crypto/chacha20/chacha.c:
  sys/crypto/chacha20/chacha.h:

    Make it possible to use the kernel implementation in libc.

PR:		182610
Reviewed by:	cem, markm
Obtained from:	OpenBSD
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D16760
2018-08-19 17:40:50 +00:00
Conrad Meyer
b8e771e97a Back out r338035 until Warner is finished churning GSoC PNP patches
I was not aware Warner was making or planning to make forward progress in
this area and have since been informed of that.

It's easy to apply/reapply when churn dies down.
2018-08-19 00:46:22 +00:00
Conrad Meyer
faa319436f Remove unused and easy to misuse PNP macro parameter
Inspired by r338025, just remove the element size parameter to the
MODULE_PNP_INFO macro entirely.  The 'table' parameter is now required to
have correct pointer (or array) type.  Since all invocations of the macro
already had this property and the emitted PNP data continues to include the
element size, there is no functional change.

Mostly done with the coccinelle 'spatch' tool:

  $ cat modpnpsize0.cocci
    @normaltables@
    identifier b,c;
    expression a,d,e;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,d,
    -sizeof(d[0]),
     e);

    @singletons@
    identifier b,c,d;
    expression a;
    declarer MODULE_PNP_INFO;
    @@
     MODULE_PNP_INFO(a,b,c,&d,
    -sizeof(d),
     1);

  $ rg -l MODULE_PNP_INFO -- sys | \
    xargs spatch --in-place --sp-file modpnpsize0.cocci

(Note that coccinelle invokes diff(1) via a PATH search and expects diff to
tolerate the -B flag, which BSD diff does not.  So I had to link gdiff into
PATH as diff to use spatch.)

Tinderbox'd (-DMAKE_JUST_KERNELS).
2018-08-19 00:22:21 +00:00
Conrad Meyer
f36e41e20b Bring in compatibility glue for libsodium
The idea is untouched upstream sources live in sys/contrib/libsodium.

sys/crypto/libsodium are support routines or compatibility headers to allow
building unmodified upstream code.

This is not yet integrated into the build system, so no functional change.
2018-08-17 00:27:56 +00:00
Xin LI
66bdf50fac libmd: Always erase context in _Final method, and when doing
it, consistently use explicit_bzero().

Update manual pages to match the behavior.

Reviewed by:	pfg, allanjude, jmg
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D16316
2018-07-20 07:01:28 +00:00
Conrad Meyer
aeb75ff3fd Remove unused variable
Reported by:	gcc
2018-07-18 04:44:11 +00:00
Conrad Meyer
2ec76e3e31 aesni(4): Add SHA2-224(-HMAC) support as well 2018-07-18 04:43:18 +00:00
Conrad Meyer
d9f87e4651 aesni(4): Add sha256 plain hash support 2018-07-18 04:37:14 +00:00
Conrad Meyer
e2982a3ed2 aesni(4): Abstract out hash/HMAC support
No functional change.

Verified with cryptocheck.
2018-07-18 04:29:44 +00:00
Conrad Meyer
1b0909d51a OpenCrypto: Convert sessions to opaque handles instead of integers
Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers.  Avoid redundancy and complexity in
individual drivers by allocating session memory in the framework and
providing it to drivers in ::newsession().

Session handles are no longer integers with information encoded in various
high bits.  Use of the CRYPTO_SESID2FOO() macros should be replaced with the
appropriate crypto_ses2foo() function on the opaque session handle.

Convert OCF drivers (in particular, cryptosoft, as well as myriad others) to
the opaque handle interface.  Discard existing session tracking as much as
possible (quick pass).  There may be additional code ripe for deletion.

Convert OCF consumers (ipsec, geom_eli, krb5, cryptodev) to handle-style
interface.  The conversion is largely mechnical.

The change is documented in crypto.9.

Inspired by
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

No objection from:	ae (ipsec portion)
Reported by:	jhb
2018-07-18 00:56:25 +00:00
Andrew Turner
a9dc38def4 Create an empty stdint.h for arm_neon.h to include.
The armv8crypto module includes arm_neon.h for the compiler intrinsic
functions. This includes the userland stdint.h file that doesn't exist in
the kernel. Fix this by providing an empty stdint.h to be used when we
include arm_neon.h.

Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D16254
2018-07-16 15:39:33 +00:00
Conrad Meyer
1245c6d1a5 Implement SHA2-224 submode of SHA2-256
Like SHA2-384:SHA2-512, SHA2-224 is simply a truncated SHA2-256 with a
different initial vector.  Add to round out the complete basic SHA2 family.
2018-07-09 07:24:05 +00:00
Conrad Meyer
590adc1bc2 Remove "HMAC" from <HASH>_HMAC_BLOCK_LEN macro names
The block size is a property of the underlying hash algorithm, and has
nothing to do with the HMAC construction.

No functional change.
2018-07-09 07:21:37 +00:00
Conrad Meyer
e50f10b5a4 aesni(4): Fix {de,en}crypt operations that allocated a buffer
aesni(4) allocates a contiguous buffer for the data it processes if the
provided input was not already virtually contiguous, and copies the input
there.  It performs encryption or decryption in-place.

r324037 removed the logic that then copied the processed data back to the
user-provided input buffer, breaking {de,enc}crypt for mbuf chains or
iovecs with more than a single descriptor.

PR:		228094 (probably, not confirmed)
Submitted by:	Sean Fagan <kithrup AT me.com>
Reported by:	Emeric POUPON <emeric.poupon AT stormshield.eu>
X-MFC-With:	324037
Security:	could result in plaintext being output by "encrypt"
		operation
2018-06-23 18:20:17 +00:00
Conrad Meyer
7d0ffa388e aesni(4): Support CRD_F_KEY_EXPLICIT OCF mode
PR:		227788
Reported by:	eadler@
2018-06-23 17:24:19 +00:00
Matt Macy
fabb4256f6 disable printing value of SKEIN_LOOP during standard out,
not useful information
2018-05-19 18:27:14 +00:00
Conrad Meyer
61590291a8 opencrypto: Integrate Chacha20 algorithm into OCF
Mostly this is a thin shim around existing code to integrate with enc_xform
and cryptosoft (+ cryptodev).

Expand the cryptodev buffer used to match that of Chacha20's native block
size as a performance enhancement for chacha20_xform_crypt_multi.
2018-03-29 04:02:50 +00:00
Conrad Meyer
2cb2ba6df8 MFV: libb2: use native calls for secure memory clearance
Drop our local patch and restore full vanilla upstream code in
contrib/libb2.

No functional change intended.  explicit_bzero() should continue to be used.

Obtained from:	libb2 b4b241a34824b51956a7866606329a065d397525
Sponsored by:	Dell EMC Isilon
2018-03-27 14:55:01 +00:00
Warner Losh
b722027307 The PNP info has to follow the module definition. Move it from just
after the array to its proper location. Otherwise, the linker.hints
file has things out of order and we associated it with whatever was
the previous module.
2018-03-25 16:56:49 +00:00
Conrad Meyer
0e33efe4e4 Import Blake2 algorithms (blake2b, blake2s) from libb2
The upstream repository is on github BLAKE2/libb2.  Files landed in
sys/contrib/libb2 are the unmodified upstream files, except for one
difference:  secure_zero_memory's contents have been replaced with
explicit_bzero() only because the previous implementation broke powerpc
link.  Preferential use of explicit_bzero() is in progress upstream, so
it is anticipated we will be able to drop this diff in the future.

sys/crypto/blake2 contains the source files needed to port libb2 to our
build system, a wrapped (limited) variant of the algorithm to match the API
of our auth_transform softcrypto abstraction, incorporation into the Open
Crypto Framework (OCF) cryptosoft(4) driver, as well as an x86 SSE/AVX
accelerated OCF driver, blake2(4).

Optimized variants of blake2 are compiled for a number of x86 machines
(anything from SSE2 to AVX + XOP).  On those machines, FPU context will need
to be explicitly saved before using blake2(4)-provided algorithms directly.
Use via cryptodev / OCF saves FPU state automatically, and use via the
auth_transform softcrypto abstraction does not use FPU.

The intent of the OCF driver is mostly to enable testing in userspace via
/dev/crypto.  ATF tests are added with published KAT test vectors to
validate correctness.

Reviewed by:	jhb, markj
Obtained from:	github BLAKE2/libb2
Differential Revision:	https://reviews.freebsd.org/D14662
2018-03-21 16:18:14 +00:00
Conrad Meyer
27cb8d849f Garbage collect unused chacha20 code
Two copies of chacha20 were imported into the tree on Apr 15 2017 (r316982)
and Apr 16 2017 (r317015).  Only the latter is actually used by anything, so
just go ahead and garbage collect the unused version while it's still only
in CURRENT.

I'm not making any judgement on which implementation is better.  If I pulled
the wrong one, feel free to swap the existing implementation out and replace
it with the other code (conforming to the API that actually gets used in
randomdev, of course).  We only need one generic implementation.

Sponsored by:	Dell EMC Isilon
2018-03-16 07:11:53 +00:00
Conrad Meyer
8475a4175f aesni(4): Stylistic/comment enhancements
Improve clarity of a comment and style(9) some areas.

No functional change.

Reported by:	markj (on review of a mostly-copied driver)
Sponsored by:	Dell EMC Isilon
2018-03-15 16:17:02 +00:00
Conrad Meyer
849ce31a82 Remove unused error return from API that cannot fail
No implementation of fpu_kern_enter() can fail, and it was causing needless
error checking boilerplate and confusion. Change the return code to void to
match reality.

(This trivial change took nine days to land because of the commit hook on
sys/dev/random.  Please consider removing the hook or otherwise lowering the
bar -- secteam never seems to have free time to review patches.)

Reported by:	Lachlan McIlroy <Lachlan.McIlroy AT isilon.com>
Reviewed by:	delphij
Approved by:	secteam (delphij)
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D14380
2018-02-23 20:15:19 +00:00
Conrad Meyer
9ea9c34ba5 ccp(4): Store IV in output buffer in GCM software fallback when requested
Apply r328361 to duplicate copy of ccr_gcm_soft in ccp(4).

Properly honor the lack of the CRD_F_IV_PRESENT flag in the GCM software
fallback case for encryption requests.
2018-01-27 07:41:31 +00:00
Conrad Meyer
844d9543dc Add ccp(4): experimental driver for AMD Crypto Co-Processor
* Registers TRNG source for random(4)
* Finds available queues, LSBs; allocates static objects
* Allocates a shared MSI-X for all queues.  The hardware does not have
  separate interrupts per queue.  Working interrupt mode driver.
* Computes SHA hashes, HMAC.  Passes cryptotest.py, cryptocheck tests.
* Does AES-CBC, CTR mode, and XTS.  cryptotest.py and cryptocheck pass.
* Support for "authenc" (AES + HMAC).  (SHA1 seems to result in
  "unaligned" cleartext inputs from cryptocheck -- which the engine
  cannot handle.  SHA2 seems to work fine.)
* GCM passes for block-multiple AAD, input lengths

Largely based on ccr(4), part of cxgbe(4).

Rough performance averages on AMD Ryzen 1950X (4kB buffer):
aesni:      SHA1: ~8300 Mb/s    SHA256: ~8000 Mb/s
ccp:               ~630 Mb/s    SHA256:  ~660 Mb/s  SHA512:  ~700 Mb/s
cryptosoft:       ~1800 Mb/s    SHA256: ~1800 Mb/s  SHA512: ~2700 Mb/s

As you can see, performance is poor in comparison to aesni(4) and even
cryptosoft (due to high setup cost).  At a larger buffer size (128kB),
throughput is a little better (but still worse than aesni(4)):

aesni:      SHA1:~10400 Mb/s    SHA256: ~9950 Mb/s
ccp:              ~2200 Mb/s    SHA256: ~2600 Mb/s  SHA512: ~3800 Mb/s
cryptosoft:       ~1750 Mb/s    SHA256: ~1800 Mb/s  SHA512: ~2700 Mb/s

AES performance has a similar story:

aesni:      4kB: ~11250 Mb/s    128kB: ~11250 Mb/s
ccp:               ~350 Mb/s    128kB:  ~4600 Mb/s
cryptosoft:       ~1750 Mb/s    128kB:  ~1700 Mb/s

This driver is EXPERIMENTAL.  You should verify cryptographic results on
typical and corner case inputs from your application against a known- good
implementation.

Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12723
2018-01-18 22:01:30 +00:00
Dimitry Andric
64e12b4140 Revert r327340, as the workaround for rep prefixes followed by .byte
directives is no longer needed after r328090.
2018-01-17 17:14:19 +00:00
Dimitry Andric
c79126f2e4 Merge ^/head r327624 through r327885. 2018-01-12 18:23:35 +00:00
Conrad Meyer
4ce147895c aesni(4): Quiesce spurious GCC 6.3.0 -Wmaybe-uninitialized warnings
Always initialize some variables GCC warns about.  They are initialized in
every path where they are used, but GCC doesn't know that.

Sponsored by:	Dell EMC Isilon
2018-01-12 06:40:58 +00:00
Dimitry Andric
2055323b2c Work around a clang 6.0.0 issue with rep prefixes followed by .byte
directives (as reported in https://bugs.llvm.org/show_bug.cgi?id=35749),
by defining the rep prefix with yet another .byte directive.

This is a temporary fix, to be reverted before merging back to head,
until upstream has a proper fix for this.
2017-12-29 12:49:24 +00:00
Pedro F. Giffuni
51369649b0 sys: further adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
2017-11-20 19:43:44 +00:00
Conrad Meyer
a480149062 aesni(4): CRYPTO_AES_NIST_GCM_16 mandates MAC
Remove some dead conditionals and add an assertion around behavior already
present in aesni_process().

Silence a few Coverity false positives.

CIDs:		1381571, 1381557
Sponsored by:	Dell EMC Isilon
2017-10-04 21:15:45 +00:00
Conrad Meyer
50cf4f8950 aesni(4): Fix GCC build
The GCC xmmintrin.h header brokenly includes mm_malloc.h unconditionally.
(The Clang version of xmmintrin.h only includes mm_malloc.h if not compiling
in standalone mode.)

Hack around GCC's broken header by defining the include guard macro ahead of
including xmmintrin.h.

Reported by:	lwhsu, jhb
Tested by:	lwhsu
Sponsored by:	Dell EMC Isilon
2017-09-29 19:56:09 +00:00
Conrad Meyer
fe182ba1d0 aesni(4): Add support for x86 SHA intrinsics
Some x86 class CPUs have accelerated intrinsics for SHA1 and SHA256.
Provide this functionality on CPUs that support it.

This implements CRYPTO_SHA1, CRYPTO_SHA1_HMAC, and CRYPTO_SHA2_256_HMAC.

Correctness: The cryptotest.py suite in tests/sys/opencrypto has been
enhanced to verify SHA1 and SHA256 HMAC using standard NIST test vectors.
The test passes on this driver.  Additionally, jhb's cryptocheck tool has
been used to compare various random inputs against OpenSSL.  This test also
passes.

Rough performance averages on AMD Ryzen 1950X (4kB buffer):
aesni:      SHA1: ~8300 Mb/s    SHA256: ~8000 Mb/s
cryptosoft:       ~1800 Mb/s    SHA256: ~1800 Mb/s

So ~4.4-4.6x speedup depending on algorithm choice.  This is consistent with
the results the Linux folks saw for 4kB buffers.

The driver borrows SHA update code from sys/crypto sha1 and sha256.  The
intrinsic step function comes from Intel under a 3-clause BSDL.[0]  The
intel_sha_extensions_sha<foo>_intrinsic.c files were renamed and lightly
modified (added const, resolved a warning or two; included the sha_sse
header to declare the functions).

[0]: https://software.intel.com/en-us/articles/intel-sha-extensions-implementations

Reviewed by:	jhb
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12452
2017-09-26 23:12:32 +00:00