Commit Graph

199 Commits

Author SHA1 Message Date
John Baldwin
55b7a0e1e7 Add support for TLS 1.3 using AES-GCM to the OCF backend for KTLS.
Reviewed by:	gallatin
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D22802
2019-12-18 01:37:00 +00:00
Andrew Gallatin
6554362c66 kTLS support for TLS 1.3
TLS 1.3 requires a few changes because 1.3 pretends to be 1.2
with a record type of application data. The "real" record type is
then included at the end of the user-supplied plaintext
data. This required adding a field to the mbuf_ext_pgs struct to
save the record type, and passing the real record type to the
sw_encrypt() ktls backend functions.

Reviewed by:	jhb, hselasky
Sponsored by:	Netflix
Differential Revision:	D21801
2019-09-27 19:17:40 +00:00
John Baldwin
c311f5ebf6 Adjust the deprecated warnings for /dev/crypto to be less noisy.
Warn when actual operations are performed instead of when sessions are
created.  The /dev/crypto engine in OpenSSL 1.0.x tries to create
sessions for all possible algorithms each time it is initialized
resulting in spurious warnings.

Reported by:	Mike Tancsa
MFC after:	3 days
Sponsored by:	Chelsio Communications
2019-08-27 21:29:37 +00:00
John Baldwin
b2e60773c6 Add kernel-side support for in-kernel TLS.
KTLS adds support for in-kernel framing and encryption of Transport
Layer Security (1.0-1.2) data on TCP sockets.  KTLS only supports
offload of TLS for transmitted data.  Key negotation must still be
performed in userland.  Once completed, transmit session keys for a
connection are provided to the kernel via a new TCP_TXTLS_ENABLE
socket option.  All subsequent data transmitted on the socket is
placed into TLS frames and encrypted using the supplied keys.

Any data written to a KTLS-enabled socket via write(2), aio_write(2),
or sendfile(2) is assumed to be application data and is encoded in TLS
frames with an application data type.  Individual records can be sent
with a custom type (e.g. handshake messages) via sendmsg(2) with a new
control message (TLS_SET_RECORD_TYPE) specifying the record type.

At present, rekeying is not supported though the in-kernel framework
should support rekeying.

KTLS makes use of the recently added unmapped mbufs to store TLS
frames in the socket buffer.  Each TLS frame is described by a single
ext_pgs mbuf.  The ext_pgs structure contains the header of the TLS
record (and trailer for encrypted records) as well as references to
the associated TLS session.

KTLS supports two primary methods of encrypting TLS frames: software
TLS and ifnet TLS.

Software TLS marks mbufs holding socket data as not ready via
M_NOTREADY similar to sendfile(2) when TLS framing information is
added to an unmapped mbuf in ktls_frame().  ktls_enqueue() is then
called to schedule TLS frames for encryption.  In the case of
sendfile_iodone() calls ktls_enqueue() instead of pru_ready() leaving
the mbufs marked M_NOTREADY until encryption is completed.  For other
writes (vn_sendfile when pages are available, write(2), etc.), the
PRUS_NOTREADY is set when invoking pru_send() along with invoking
ktls_enqueue().

A pool of worker threads (the "KTLS" kernel process) encrypts TLS
frames queued via ktls_enqueue().  Each TLS frame is temporarily
mapped using the direct map and passed to a software encryption
backend to perform the actual encryption.

(Note: The use of PHYS_TO_DMAP could be replaced with sf_bufs if
someone wished to make this work on architectures without a direct
map.)

KTLS supports pluggable software encryption backends.  Internally,
Netflix uses proprietary pure-software backends.  This commit includes
a simple backend in a new ktls_ocf.ko module that uses the kernel's
OpenCrypto framework to provide AES-GCM encryption of TLS frames.  As
a result, software TLS is now a bit of a misnomer as it can make use
of hardware crypto accelerators.

Once software encryption has finished, the TLS frame mbufs are marked
ready via pru_ready().  At this point, the encrypted data appears as
regular payload to the TCP stack stored in unmapped mbufs.

ifnet TLS permits a NIC to offload the TLS encryption and TCP
segmentation.  In this mode, a new send tag type (IF_SND_TAG_TYPE_TLS)
is allocated on the interface a socket is routed over and associated
with a TLS session.  TLS records for a TLS session using ifnet TLS are
not marked M_NOTREADY but are passed down the stack unencrypted.  The
ip_output_send() and ip6_output_send() helper functions that apply
send tags to outbound IP packets verify that the send tag of the TLS
record matches the outbound interface.  If so, the packet is tagged
with the TLS send tag and sent to the interface.  The NIC device
driver must recognize packets with the TLS send tag and schedule them
for TLS encryption and TCP segmentation.  If the the outbound
interface does not match the interface in the TLS send tag, the packet
is dropped.  In addition, a task is scheduled to refresh the TLS send
tag for the TLS session.  If a new TLS send tag cannot be allocated,
the connection is dropped.  If a new TLS send tag is allocated,
however, subsequent packets will be tagged with the correct TLS send
tag.  (This latter case has been tested by configuring both ports of a
Chelsio T6 in a lagg and failing over from one port to another.  As
the connections migrated to the new port, new TLS send tags were
allocated for the new port and connections resumed without being
dropped.)

ifnet TLS can be enabled and disabled on supported network interfaces
via new '[-]txtls[46]' options to ifconfig(8).  ifnet TLS is supported
across both vlan devices and lagg interfaces using failover, lacp with
flowid enabled, or lacp with flowid enabled.

Applications may request the current KTLS mode of a connection via a
new TCP_TXTLS_MODE socket option.  They can also use this socket
option to toggle between software and ifnet TLS modes.

In addition, a testing tool is available in tools/tools/switch_tls.
This is modeled on tcpdrop and uses similar syntax.  However, instead
of dropping connections, -s is used to force KTLS connections to
switch to software TLS and -i is used to switch to ifnet TLS.

Various sysctls and counters are available under the kern.ipc.tls
sysctl node.  The kern.ipc.tls.enable node must be set to true to
enable KTLS (it is off by default).  The use of unmapped mbufs must
also be enabled via kern.ipc.mb_use_ext_pgs to enable KTLS.

KTLS is enabled via the KERN_TLS kernel option.

This patch is the culmination of years of work by several folks
including Scott Long and Randall Stewart for the original design and
implementation; Drew Gallatin for several optimizations including the
use of ext_pgs mbufs, the M_NOTREADY mechanism for TLS records
awaiting software encryption, and pluggable software crypto backends;
and John Baldwin for modifications to support hardware TLS offload.

Reviewed by:	gallatin, hselasky, rrs
Obtained from:	Netflix
Sponsored by:	Netflix, Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21277
2019-08-27 00:01:56 +00:00
John Baldwin
87210a0c6d Stop using des_cblock * for arguments to DES functions.
This amounts to a char ** since it is a char[8] *.  Evil casts mostly
resolved the fact that what was actually passed in were plain char *.
Instead, change the DES functions to use 'unsigned char *' for keys
and for input and output buffers.

Reviewed by:	cem, imp
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21389
2019-08-26 17:25:07 +00:00
John Baldwin
821fe3d3a4 Use 'const' for keys and IVs passed to software encryption algorithms.
Specifically, use 'const' for the key passed to the 'setkey' method
and 'const' for the 'iv' passed to the 'reinit' method.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21347
2019-08-22 00:02:08 +00:00
Xin LI
0ed1d6fb00 Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib,
with an eventual goal to convert all legacl zlib callers to the new zlib
version:

 * Move generic zlib shims that are not specific to zlib 1.0.4 to
   sys/dev/zlib.
 * Connect new zlib (1.2.11) to the zlib kernel module, currently built
   with Z_SOLO.
 * Prefix the legacy zlib (1.0.4) with 'zlib104_' namespace.
 * Convert sys/opencrypto/cryptodeflate.c to use new zlib.
 * Remove bundled zlib 1.2.3 from ZFS and adapt it to new zlib and make
   it depend on the zlib module.
 * Fix Z_SOLO build of new zlib.

PR:		229763
Submitted by:	Yoshihiro Ota <ota j email ne jp>
Reviewed by:	markm (sys/dev/zlib/zlib_kmod.c)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D19706
2019-08-01 06:35:33 +00:00
John Baldwin
a0c4047d4d Move declaration of warninterval out from under COMPAT_FREEBSD32.
This fixes builds of kernels without COMPAT_FREEBSD32.

Reported by:	tinderbox
MFC after:	1 month
2019-06-11 23:28:07 +00:00
John Baldwin
0f70218343 Make the warning intervals for deprecated crypto algorithms tunable.
New sysctl/tunables can now set the interval (in seconds) between
rate-limited crypto warnings.  The new sysctls are:
- kern.cryptodev_warn_interval for /dev/crypto
- net.inet.ipsec.crypto_warn_interval for IPsec
- kern.kgssapi_warn_interval for KGSSAPI

Reviewed by:	cem
MFC after:	1 month
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20555
2019-06-11 23:00:55 +00:00
John Baldwin
5e35041990 Add warnings to /dev/crypto for deprecated algorithms.
These algorithms are deprecated algorithms that will have no in-kernel
consumers in FreeBSD 13.  Specifically, deprecate the following
algorithms:
- ARC4
- Blowfish
- CAST128
- DES
- 3DES
- MD5-HMAC
- Skipjack

MFC after:	1 month
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20554
2019-06-10 19:26:57 +00:00
Xin LI
a49818787d cryptodeflate: Drop z_stream zbuf.state->dummy from SDT probe.
For older versions of zlib, dummy was a workaround for compilers that do not
handle opaque type definition well; on FreeBSD, it's representing a value
that is not really useful for monitoring purposes, and the field would be gone
in newer zlib versions.

PR:		229763
Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
Differential Revision:	https://reviews.freebsd.org/D20222
2019-05-24 02:44:15 +00:00
John Baldwin
8ccf3d974f Don't panic for empty CCM requests.
A request to encrypt an empty payload without any AAD is unusual, but
it is defined behavior.  Removing this assertion removes a panic and
instead returns the correct tag for an empty buffer.

Reviewed by:	cem, sef
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20043
2019-04-24 23:27:39 +00:00
Sean Eric Fagan
f42230d856 Fix another bug introduced during the review process of r344140:
the tag wasn't being computed properly due to chaning a >= comparison
to an == comparison.

Specifically:  CBC-MAC encodes the length of the authorization data
into the the stream to be encrypted/hashed.  For short data, this is
two bytes (big-endian 16 bit value); for larger data, it's 6 bytes
(a prefix of 0xff, 0xfe, followed by a 32-bit big-endian length).  And
there's a larger size, which is 10 bytes.  These extra bytes weren't
being accounted for with the post-review code.  The other bit that then came
into play was that OCF only calls the Update code with blksiz=16, which
meant that I had to ignore the length variable.  (It also means that it
can't be called with a single buffer containing the AAD and payload;
however, OCF doesn't do this for the software-only algorithsm.)

I tested with this script:

ALG=aes-ccm
DEV=soft

for aad in 0 1 2 3 4 14 16 24 30 32 34 36 1020
do
        for dln in 16 32 1024 2048 10240
        do
                echo "Testing AAD length ${aad} data length ${dln}"
                /root/cryptocheck -A ${aad} -a ${ALG} -d ${DEV} ${dln}
        done
done

Reviewed by:	cem
Sponsored by:	iXsystems Inc.
2019-02-25 19:14:16 +00:00
Sean Eric Fagan
1357a3bc19 Fix another issue from r344141, having to do with size of a shift amount.
This did not show up in my testing.

Differential Revision:	https://reviews.freebsd.org/D18592
2019-02-15 04:15:43 +00:00
Sean Eric Fagan
72309077eb Pasting in a source control line missed the last quote. Fixed. 2019-02-15 04:01:59 +00:00
Sean Eric Fagan
507281e55e Add AES-CCM encryption, and plumb into OCF.
This commit essentially has three parts:

* Add the AES-CCM encryption hooks.  This is in and of itself fairly small,
as there is only a small difference between CCM and the other ICM-based
algorithms.
* Hook the code into the OpenCrypto framework.  This is the bulk of the
changes, as the algorithm type has to be checked for, and the differences
between it and GCM dealt with.
* Update the cryptocheck tool to be aware of it.  This is invaluable for
confirming that the code works.

This is a software-only implementation, meaning that the performance is very
low.

Sponsored by:	iXsystems Inc.
Differential Revision:	https://reviews.freebsd.org/D19090
2019-02-15 03:53:03 +00:00
Sean Eric Fagan
a99bc4c3eb Add CBC-MAC authentication.
This adds the CBC-MAC code to the kernel, but does not hook it up to
anything (that comes in the next commit).

https://tools.ietf.org/html/rfc3610 describes the algorithm.

Note that this is a software-only implementation, which means it is
fairly slow.

Sponsored by:   iXsystems Inc
Differential Revision:  https://reviews.freebsd.org/D18592
2019-02-15 03:46:39 +00:00
Marius Strobl
345c692d18 As struct cryptop is wrapped in #ifdef _KERNEL, userland doesn't
need to drag in <sys/_task.h> either.
2019-02-10 21:27:03 +00:00
Andrey V. Elsukov
6062df0326 Plug memory leak for AES_*_NIST_GMAC algorithms.
swcr_newsession() allocates sw_ictx for these algorithms, thus we need
to free() it in swcr_freesession().

PR:		233907
MFC after:	1 week
2018-12-13 08:59:51 +00:00
Matt Macy
ff2038a9bf Generalize AES iov optimization
Right now, aesni_cipher_alloc does a bit of special-casing
for CRYPTO_F_IOV, to not do any allocation if the first uio
is large enough for the requested size. While working on ZFS
crypto port, I ran into horrible performance because the code
uses scatter-gather, and many of the times the data to encrypt
was in the second entry. This code looks through the list, and
tries to see if there is a single uio that can contain the
requested data, and, if so, uses that.

This has a slight impact on the current consumers, in that the
check is a little more complicated for the ones that use
CRYPTO_F_IOV -- but none of them meet the criteria for testing
more than one.

Submitted by:	sef at ixsystems.com
Reviewed by:	cem@
MFC after:	3 days
Sponsored by:	iX Systems
Differential Revision:	https://reviews.freebsd.org/D18522
2018-12-13 04:40:53 +00:00
John Baldwin
174a501466 Add sha224 to the authctx union.
MFC after:	2 months
Sponsored by:	Chelsio Communications
2018-10-23 18:07:37 +00:00
Sean Eric Fagan
a7fcb1afcb Add per-session locking to cryptosoft (swcr).
As part of ZFS Crypto, I started getting a series of panics when I did not
have AESNI loaded.  Adding locking fixed it, and I concluded that the
Reinit function altered the AES key schedule.  This locking is not as
fine-grained as it could be (AESNI uses per-cpu locking), but
it's minimally invasive.

Sponsored by: iXsystems Inc
Reviewed by: cem, mav
Approved by: re (gjb), mav (mentor)
Differential Revision: https://reviews.freebsd.org/D17307
2018-09-26 20:23:12 +00:00
Conrad Meyer
9ebbebe4f7 cryptosoft: Reduce generality of supported algorithm composition
Fix a regression introduced in r336439.

Rather than allowing any linked list of algorithms, allow at most two
(typically, some combination of encrypt and/or MAC).  Removes a WAITOK
malloc in an unsleepable context (classic LOR) by placing both software
algorithm contexts within the OCF-managed session object.

Tested with 'cryptocheck -a all -d cryptosoft0', which includes some
encrypt-and-MAC modes.

PR:		230304
Reported by:	sef@
2018-08-17 04:40:01 +00:00
Conrad Meyer
25b7033b73 crypto(4): Add cryptosoft, cryptodev support for Poly-1305 2018-08-17 00:31:06 +00:00
Conrad Meyer
01d5de8fca Add xform-conforming auth_hash wrapper for Poly-1305
The wrapper is a thin shim around libsodium's Poly-1305 implementation.  For
now, we just use the C algorithm and do not attempt to build the
SSE-optimized variant for x86 processors.

The algorithm support has not yet been plumbed through cryptodev, or added
to cryptosoft.
2018-08-17 00:30:04 +00:00
Alan Somers
6040822c4e Make timespecadd(3) and friends public
The timespecadd(3) family of macros were imported from NetBSD back in
r35029. However, they were initially guarded by #ifdef _KERNEL. In the
meantime, we have grown at least 28 syscalls that use timespecs in some
way, leading many programs both inside and outside of the base system to
redefine those macros. It's better just to make the definitions public.

Our kernel currently defines two-argument versions of timespecadd and
timespecsub.  NetBSD, OpenBSD, and FreeDesktop.org's libbsd, however, define
three-argument versions.  Solaris also defines a three-argument version, but
only in its kernel.  This revision changes our definition to match the
common three-argument version.

Bump _FreeBSD_version due to the breaking KPI change.

Discussed with:	cem, jilles, ian, bde
Differential Revision:	https://reviews.freebsd.org/D14725
2018-07-30 15:46:40 +00:00
Conrad Meyer
1b0909d51a OpenCrypto: Convert sessions to opaque handles instead of integers
Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers.  Avoid redundancy and complexity in
individual drivers by allocating session memory in the framework and
providing it to drivers in ::newsession().

Session handles are no longer integers with information encoded in various
high bits.  Use of the CRYPTO_SESID2FOO() macros should be replaced with the
appropriate crypto_ses2foo() function on the opaque session handle.

Convert OCF drivers (in particular, cryptosoft, as well as myriad others) to
the opaque handle interface.  Discard existing session tracking as much as
possible (quick pass).  There may be additional code ripe for deletion.

Convert OCF consumers (ipsec, geom_eli, krb5, cryptodev) to handle-style
interface.  The conversion is largely mechnical.

The change is documented in crypto.9.

Inspired by
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

No objection from:	ae (ipsec portion)
Reported by:	jhb
2018-07-18 00:56:25 +00:00
Conrad Meyer
2e08e39ff5 OCF: Add a typedef for session identifiers
No functional change.

This should ease the transition from an integer session identifier model to
an opaque pointer model.
2018-07-13 23:46:07 +00:00
Conrad Meyer
c4729f6e89 OCF: Add plain hash modes
In part, to support OpenSSL's use of cryptodev, which puts the HMAC pieces
in software and only offloads the raw hash primitive.

The following cryptodev identifiers are added:

 * CRYPTO_RIPEMD160 (not hooked up)
 * CRYPTO_SHA2_224
 * CRYPTO_SHA2_256
 * CRYPTO_SHA2_384
 * CRYPTO_SHA2_512

The plain SHA1 and 2 hashes are plumbed through cryptodev (feels like there
is a lot of redundancy here...) and cryptosoft.

This adds new auth_hash implementations for the plain hashes, as well as
SHA1 (which had a cryptodev.h identifier, but no implementation).

Add plain SHA 1 and 2 hash tests to the cryptocheck tool.

Motivation stems from John Baldwin's earlier OCF email,
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .
2018-07-09 07:28:13 +00:00
Conrad Meyer
c97f39ce17 OCF: Add CRYPTO_SHA2_224_HMAC mode
Round out the complete set of basic SHA2 HMAC modes with SHA2-224.

Support is added to the cryptocheck test tool.
2018-07-09 07:26:12 +00:00
Conrad Meyer
590adc1bc2 Remove "HMAC" from <HASH>_HMAC_BLOCK_LEN macro names
The block size is a property of the underlying hash algorithm, and has
nothing to do with the HMAC construction.

No functional change.
2018-07-09 07:21:37 +00:00
Conrad Meyer
179b21e8b1 cryptosoft: Do not exceed crd_len around *crypt_multi
When a caller passes in a uio or mbuf chain that is longer than crd_len, in
tandem with a transform that supports the multi-block interface,
swcr_encdec() would process the entire mbuf or uio instead of just the
portion indicated by crd_len (+ crd_skip).

De/encryption are performed in-place, so this would trash subsequent uio or
mbuf contents.

This was introduced in r331639 (mea culpa).  It only affects the
{de,en}crypt_multi() family of interfaces.  That interface only has one
consumer transform in-tree (for now): Chacha20.

PR:		227605
Submitted by:	Valentin Vergez <valentin.vergez AT stormshield.eu>
2018-04-19 15:24:21 +00:00
Brooks Davis
6469bdcdb6 Move most of the contents of opt_compat.h to opt_global.h.
opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
closer to "just about everywhere" than "only some files" per the
guidance in sys/conf/options.

Keep COMPAT_LINUX32 in opt_compat.h as it is confined to a subset of
sys/compat/linux/*.c.  A fake _COMPAT_LINUX option ensure opt_compat.h
is created on all architectures.

Move COMPAT_LINUXKPI to opt_dontuse.h as it is only used to control the
set of compiled files.

Reviewed by:	kib, cem, jhb, jtl
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D14941
2018-04-06 17:35:35 +00:00
Conrad Meyer
5d7ae54a5d cryptosoft: Remove a dead store
Introduced in r331639 by removing an instance of undefined behavior.

While we're here, the variable scope can be entirely moved inside the loop.

Reported by:	Coverity
CID:		1387985
Sponsored by:	Dell EMC Isilon
2018-04-03 22:11:39 +00:00
Conrad Meyer
61590291a8 opencrypto: Integrate Chacha20 algorithm into OCF
Mostly this is a thin shim around existing code to integrate with enc_xform
and cryptosoft (+ cryptodev).

Expand the cryptodev buffer used to match that of Chacha20's native block
size as a performance enhancement for chacha20_xform_crypt_multi.
2018-03-29 04:02:50 +00:00
Conrad Meyer
2f1f9ccea7 opencrypto: Add mechanism to pass multiple crypto blocks to some ciphers
xforms that support processing of multiple blocks at a time (to support more
efficient modes, for example) can define the encrypt_ and decrypt_multi
interfaces.  If these interfaces are not present, the generic cryptosoft
code falls back on the block-at-a-time encrypt/decrypt interfaces.

Stream ciphers may support arbitrarily sized inputs (equivalent to an input
block size of 1 byte) but may be more efficient if a larger block is passed.

Sponsored by:	Dell EMC Isilon
2018-03-27 17:58:00 +00:00
Conrad Meyer
289b9798be OCF: CRYPTDEB(): Enhance to allow formatted logging
Sponsored by:	Dell EMC Isilon
2018-03-26 22:31:29 +00:00
Conrad Meyer
19d0de8d64 cryptodev: Match intent for enc_xform ciphers with blocksize != ivsize
No functional change for Skipjack, AES-ICM, Blowfish, CAST-128, Camellia,
DES3, Rijndael128, DES.  All of these have identical IV and blocksizes
declared in the associated enc_xform.

Functional changes for:
  * AES-GCM: block len of 1, IV len of 12
  * AES-XTS: block len of 16, IV len of 8
  * NULL: block len of 4, IV len of 0

For these, it seems like the IV specified in the enc_xform is correct (and
the blocksize used before was wrong).

Additionally, the not-yet-OCFed cipher Chacha20 has a logical block length
of 1 byte, and a 16 byte IV + nonce.

Rationalize references to IV lengths to refer to the declared ivsize, rather
than declared blocksize.

Sponsored by:	Dell EMC Isilon
2018-03-26 20:30:07 +00:00
Conrad Meyer
0e33efe4e4 Import Blake2 algorithms (blake2b, blake2s) from libb2
The upstream repository is on github BLAKE2/libb2.  Files landed in
sys/contrib/libb2 are the unmodified upstream files, except for one
difference:  secure_zero_memory's contents have been replaced with
explicit_bzero() only because the previous implementation broke powerpc
link.  Preferential use of explicit_bzero() is in progress upstream, so
it is anticipated we will be able to drop this diff in the future.

sys/crypto/blake2 contains the source files needed to port libb2 to our
build system, a wrapped (limited) variant of the algorithm to match the API
of our auth_transform softcrypto abstraction, incorporation into the Open
Crypto Framework (OCF) cryptosoft(4) driver, as well as an x86 SSE/AVX
accelerated OCF driver, blake2(4).

Optimized variants of blake2 are compiled for a number of x86 machines
(anything from SSE2 to AVX + XOP).  On those machines, FPU context will need
to be explicitly saved before using blake2(4)-provided algorithms directly.
Use via cryptodev / OCF saves FPU state automatically, and use via the
auth_transform softcrypto abstraction does not use FPU.

The intent of the OCF driver is mostly to enable testing in userspace via
/dev/crypto.  ATF tests are added with published KAT test vectors to
validate correctness.

Reviewed by:	jhb, markj
Obtained from:	github BLAKE2/libb2
Differential Revision:	https://reviews.freebsd.org/D14662
2018-03-21 16:18:14 +00:00
Conrad Meyer
5fbc5b5a3c cryptosoft(4): Zero plain hash contexts, too
An OCF-naive user program could use these primitives to implement HMAC, for
example.  This would make the freed context sensitive data.

Probably other bzeros in this file should be explicit_bzeros as well.
Future work.

Reviewed by:	jhb, markj
Differential Revision:	https://reviews.freebsd.org/D14662 (minor part of a larger work)
2018-03-21 16:12:07 +00:00
John Baldwin
5425750f03 Move per-operation data out of the csession structure.
Create a struct cryptop_data which contains state needed for a single
symmetric crypto operation and move that state out of the session. This
closes a race with the CRYPTO_F_DONE flag that can result in use after
free.

While here, remove the 'cse->error' member.  It was just a copy of
'crp->crp_etype' and cryptodev_op() and cryptodev_aead() checked both
'crp->crp_etype' and 'cse->error'.  Similarly, do not check for an
error from mtx_sleep() since it is not used with PCATCH or a timeout
so cannot fail with an error.

PR:		218597
Reviewed by:	kib
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13928
2018-01-26 23:21:50 +00:00
John Baldwin
a296c71966 Split crp_buf into a union.
This adds explicit crp_mbuf and crp_uio pointers of the right type to
replace casts of crp_buf.  This does not sweep through changing existing
code, but new code should use the correct fields instead of casts.

Reviewed by:	kib
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13927
2018-01-16 19:41:18 +00:00
John Baldwin
eda08c83c7 Change the type of 'crp_opaque' from caddr_t to void *.
Opaque pointers should be void *.  Note that this does not go through
the tree removing all of the now-unnecessary casts.

Reviewed by:	kib
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13848
2018-01-11 18:09:24 +00:00
John Baldwin
eb22dfe39b Axe tmp_iv from the cryptodev session structure.
Just copyin the IV into the crypto descriptor directly.  This avoids
copying the IV twice for each operation.

Reviewed by:	kib
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13847
2018-01-11 18:07:21 +00:00
John Baldwin
776a2127ef Flesh out static dtrace probes for /dev/crypto ioctl errors.
In particular, no probes were present for AEAD requests, but also for
some other error cases in other ioctl requests.

MFC after:	2 weeks
Sponsored by:	Chelsio Communications
2018-01-11 00:22:24 +00:00
Fabien Thomas
de2b2c908a Fix uninitialized crp_retw_id when using asynchronous crypto drivers
with defered callbacks.

Submitted by:	emeric.poupon@stormshield.eu
Reported by:	mav@
Reviewed by:	fabient@
2018-01-08 13:43:12 +00:00
John Baldwin
8bbeea2b1d Remove a redunant check. 2017-12-30 03:08:49 +00:00
Alexander Kabaev
151ba7933a Do pass removing some write-only variables from the kernel.
This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385
2017-12-25 04:48:39 +00:00
Warner Losh
3a7d67e741 We don't need both _STAND and _STANDALONE. There's more places that
use _STANDALONE, so change the former to the latter.

Sponsored by: Netflix
2017-12-02 00:07:09 +00:00
Fabien Thomas
39bbca6ffd crypto(9) is called from ipsec in CRYPTO_F_CBIFSYNC mode. This is working
fine when a lot of different flows to be ciphered/deciphered are involved.

However, when a software crypto driver is used, there are
situations where we could benefit from making crypto(9) multi threaded:
- a single flow is to be ciphered: only one thread is used to cipher it,
- a single ESP flow is to be deciphered: only one thread is used to
decipher it.

The idea here is to call crypto(9) using a new mode (CRYPTO_F_ASYNC) to
dispatch the crypto jobs on multiple threads, if the underlying crypto
driver is working in synchronous mode.

Another flag is added (CRYPTO_F_ASYNC_KEEPORDER) to make crypto(9)
dispatch the crypto jobs in the order they are received (an additional
queue/thread is used), so that the packets are reinjected in the network
using the same order they were posted.

A new sysctl net.inet.ipsec.async_crypto can be used to activate
this new behavior (disabled by default).

Submitted by:	Emeric Poupon <emeric.poupon@stormshield.eu>
Reviewed by:	ae, jmg, jhb
Differential Revision:    https://reviews.freebsd.org/D10680
Sponsored by:	Stormshield
2017-11-03 10:27:22 +00:00