Commit Graph

231 Commits

Author SHA1 Message Date
jhb
ff11fae515 Increment the correct pointer when a crypto buffer spans an mbuf or iovec.
When a crypto_cursor_copyback() request spanned multiple mbufs or
iovecs, the pointer into the mbuf/iovec was incremented instead of the
pointer into the source buffer being copied from.

PR:		246737
Reported by:	Jenkins, ZFS test suite
Sponsored by:	Netflix
2020-05-29 05:41:21 +00:00
jhb
f0a6418f59 Add a sysctl knob to use separate output buffers for /dev/crypto.
This is a testing aid to permit using testing a driver's support of
separate output buffers via cryptocheck.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24545
2020-05-25 22:21:09 +00:00
jhb
a07ba3794f Export the _kern_crypto sysctl node from crypto.c.
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24545
2020-05-25 22:18:33 +00:00
jhb
8f001f91aa Add support for optional separate output buffers to in-kernel crypto.
Some crypto consumers such as GELI and KTLS for file-backed sendfile
need to store their output in a separate buffer from the input.
Currently these consumers copy the contents of the input buffer into
the output buffer and queue an in-place crypto operation on the output
buffer.  Using a separate output buffer avoids this copy.

- Create a new 'struct crypto_buffer' describing a crypto buffer
  containing a type and type-specific fields.  crp_ilen is gone,
  instead buffers that use a flat kernel buffer have a cb_buf_len
  field for their length.  The length of other buffer types is
  inferred from the backing store (e.g. uio_resid for a uio).
  Requests now have two such structures: crp_buf for the input buffer,
  and crp_obuf for the output buffer.

- Consumers now use helper functions (crypto_use_*,
  e.g. crypto_use_mbuf()) to configure the input buffer.  If an output
  buffer is not configured, the request still modifies the input
  buffer in-place.  A consumer uses a second set of helper functions
  (crypto_use_output_*) to configure an output buffer.

- Consumers must request support for separate output buffers when
  creating a crypto session via the CSP_F_SEPARATE_OUTPUT flag and are
  only permitted to queue a request with a separate output buffer on
  sessions with this flag set.  Existing drivers already reject
  sessions with unknown flags, so this permits drivers to be modified
  to support this extension without requiring all drivers to change.

- Several data-related functions now have matching versions that
  operate on an explicit buffer (e.g. crypto_apply_buf,
  crypto_contiguous_subsegment_buf, bus_dma_load_crp_buf).

- Most of the existing data-related functions operate on the input
  buffer.  However crypto_copyback always writes to the output buffer
  if a request uses a separate output buffer.

- For the regions in input/output buffers, the following conventions
  are followed:
  - AAD and IV are always present in input only and their
    fields are offsets into the input buffer.
  - payload is always present in both buffers.  If a request uses a
    separate output buffer, it must set a new crp_payload_start_output
    field to the offset of the payload in the output buffer.
  - digest is in the input buffer for verify operations, and in the
    output buffer for compute operations.  crp_digest_start is relative
    to the appropriate buffer.

- Add a crypto buffer cursor abstraction.  This is a more general form
  of some bits in the cryptosoft driver that tried to always use uio's.
  However, compared to the original code, this avoids rewalking the uio
  iovec array for requests with multiple vectors.  It also avoids
  allocate an iovec array for mbufs and populating it by instead walking
  the mbuf chain directly.

- Update the cryptosoft(4) driver to support separate output buffers
  making use of the cursor abstraction.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24545
2020-05-25 22:12:04 +00:00
jhb
938b10d581 Correct the minimum key length for Camellia to 16 bytes (128 bits).
MFC after:	1 week
2020-05-22 17:21:22 +00:00
jhb
0c1410ac3b Improve support for stream ciphers in the software encryption interface.
Add a 'native_blocksize' member to 'struct enc_xform' that ciphers can
use if they support a partial final block.  This is particular useful
for stream ciphers, but can also apply to other ciphers.  cryptosoft
will only pass in native blocks to the encrypt and decrypt hooks.  For
the final partial block, 'struct enc_xform' now has new
encrypt_last/decrypt_last hooks which accept the length of the final
block.  The multi_block methods are also retired.

Mark AES-ICM (AES-CTR) as a stream cipher.  This has some interesting
effects on IPsec in that FreeBSD can now properly receive all packets
sent by Linux when using AES-CTR, but FreeBSD can no longer
interoperate with OpenBSD and older verisons of FreeBSD which assume
AES-CTR packets have a payload padded to a 16-byte boundary.  Kornel
has offered to work on a patch to add a compatiblity sysctl to enforce
additional padding for AES-CTR in esp_output to permit compatibility
with OpenBSD and older versions of FreeBSD.

AES-XTS continues to use a block size of a single AES block length.
It is possible to adjust it to support partial final blocks by
implementing cipher text stealing via encrypt_last/decrypt_last hooks,
but I have not done so.

Reviewed by:	cem (earlier version)
Tested by:	Kornel Dulęba <mindal@semihalf.com> (AES-CTR with IPsec)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24906
2020-05-22 16:29:09 +00:00
jhb
52934e620a Fix libstand build breakage after r361298.
- Use enc_xform_aes_xts.setkey() directly instead of duplicating the code
  now that it no longer calls malloc().
- Rather than bringing back all of xform_userland.h, add a conditional
  #include of <stand.h> to xform_enc.h.
- Update calls to encrypt/decrypt callbacks in enc_xform_aes_xts for
  separate input/output pointers.

Pointy hat to:	jhb
2020-05-20 22:25:41 +00:00
jhb
4fcb017f63 Various cleanups to the software encryption transform interface.
- Consistently use 'void *' for key schedules / key contexts instead
  of a mix of 'caddr_t', 'uint8_t *', and 'void *'.

- Add a ctxsize member to enc_xform similar to what auth transforms use
  and require callers to malloc/zfree the context.  The setkey callback
  now supplies the caller-allocated context pointer and the zerokey
  callback is removed.  Callers now always use zfree() to ensure
  key contexts are zeroed.

- Consistently use C99 initializers for all statically-initialized
  instances of 'struct enc_xform'.

- Change the encrypt and decrypt functions to accept separate in and
  out buffer pointers.  Almost all of the backend crypto functions
  already supported separate input and output buffers and this makes
  it simpler to support separate buffers in OCF.

- Remove xform_userland.h shim to permit transforms to be compiled in
  userland.  Transforms no longer call malloc/free directly.

Reviewed by:	cem (earlier version)
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24855
2020-05-20 21:21:01 +00:00
jhb
38dd4e2948 Trim a few more things I missed from xform_enc.h.
An extern declaration for the now-removed Blowfish encryption
transform, and an include of the DES header.
2020-05-13 18:36:02 +00:00
jhb
3872b395c0 Remove MD5 HMAC from OCF.
There are no in-kernel consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24775
2020-05-11 22:08:08 +00:00
kp
3618e3148f opencrypto: Add missing ioctl exit SDTs
The opencrypto ioctl code has very useful probe points at the various exit
points. These allow us to figure out exactly why a request failed. However, a
few paths did not have these probe points. Add them here.

Reviewed by:	jhb
2020-05-11 21:42:19 +00:00
jhb
489f2f2aa1 Remove support for DES and Triple DES from OCF.
It no longer has any in-kernel consumers via OCF.  smbfs still uses
single DES directly, so sys/crypto/des remains for that use case.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24773
2020-05-11 21:34:29 +00:00
jhb
d095cb1c79 Remove support for the Blowfish algorithm from OCF.
It no longer has any in-kernel consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24772
2020-05-11 21:24:05 +00:00
jhb
d975aa5318 Remove support for the ARC4 algorithm from OCF.
There are no longer any in-kernel consumers.  The software
implementation was also a non-functional stub.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24771
2020-05-11 21:17:08 +00:00
jhb
bf8eb2dd71 Remove support for keyed MD5 and SHA1 authentication hashes.
They no longer have any in-tree consumers.  Note that these are a
different from MD5-HMAC and SHA1-HMAC and were only used with IPsec.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24770
2020-05-11 21:04:59 +00:00
jhb
12d0af4853 Remove support for the skipjack encryption algorithm.
This was removed from IPsec in r286100 and no longer has any in-tree
consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24769
2020-05-11 20:54:59 +00:00
jhb
a2167414c4 Remove support for the cast128 encryption algorithm.
It no longer has any in-tree consumers.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24768
2020-05-11 20:52:18 +00:00
jhb
fc24e5daa6 Remove incomplete support for plain MD5 from OCF.
Although a few drivers supported this algorithm, there were never any
in-kernel consumers.  cryptosoft and cryptodev never supported it,
and there was not a software xform auth_hash for it.

Reviewed by:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24767
2020-05-11 20:40:30 +00:00
jhb
81bd4d44b5 Use a lookup table of algorithm types for alg_is_* helpers.
This makes it easier to maintain these functions as algorithms are
added or removed.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24668
2020-05-04 22:31:38 +00:00
jhb
614a22abcb Actually remove support for Triple DES, not just the warning.
Missed in r360562.
2020-05-02 17:18:32 +00:00
jhb
b345efd4ae Remove support for the algorithms deprecated in r348876.
This removes support for the following algorithms:
- ARC4
- Blowfish
- CAST128
- DES
- 3DES
- MD5-HMAC
- Skipjack

Since /dev/crypto no longer supports 3DES, stop testing the 3DES KAT
vectors in cryptotest.py.

Reviewed by:	cem (previous version)
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D24346
2020-05-02 14:20:32 +00:00
jhb
5ddc1a3518 Retire the CRYPTO_F_IV_GENERATE flag.
The sole in-tree user of this flag has been retired, so remove this
complexity from all drivers.  While here, add a helper routine drivers
can use to read the current request's IV into a local buffer.  Use
this routine to replace duplicated code in nearly all drivers.

Reviewed by:	cem
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D24450
2020-04-20 22:24:49 +00:00
jhb
54906d4f65 Avoid checking pointers that are never NULL.
Coverity noted that cod pointer is always non-NULL at the end of
cryptodev_aead().  While here, fix cryptodev_op() to match by making
one earlier failure case before cod and crp are allocated just return
directly.

CID:		1422185
Reported by:	Coverity
2020-04-02 17:22:16 +00:00
jhb
512a48dee1 Remove duplicate conditional.
CID:		1422192
Reported by:	Coverity
2020-04-02 16:55:28 +00:00
jhb
ddcef18974 Refactor driver and consumer interfaces for OCF (in-kernel crypto).
- The linked list of cryptoini structures used in session
  initialization is replaced with a new flat structure: struct
  crypto_session_params.  This session includes a new mode to define
  how the other fields should be interpreted.  Available modes
  include:

  - COMPRESS (for compression/decompression)
  - CIPHER (for simply encryption/decryption)
  - DIGEST (computing and verifying digests)
  - AEAD (combined auth and encryption such as AES-GCM and AES-CCM)
  - ETA (combined auth and encryption using encrypt-then-authenticate)

  Additional modes could be added in the future (e.g. if we wanted to
  support TLS MtE for AES-CBC in the kernel we could add a new mode
  for that.  TLS modes might also affect how AAD is interpreted, etc.)

  The flat structure also includes the key lengths and algorithms as
  before.  However, code doesn't have to walk the linked list and
  switch on the algorithm to determine which key is the auth key vs
  encryption key.  The 'csp_auth_*' fields are always used for auth
  keys and settings and 'csp_cipher_*' for cipher.  (Compression
  algorithms are stored in csp_cipher_alg.)

- Drivers no longer register a list of supported algorithms.  This
  doesn't quite work when you factor in modes (e.g. a driver might
  support both AES-CBC and SHA2-256-HMAC separately but not combined
  for ETA).  Instead, a new 'crypto_probesession' method has been
  added to the kobj interface for symmteric crypto drivers.  This
  method returns a negative value on success (similar to how
  device_probe works) and the crypto framework uses this value to pick
  the "best" driver.  There are three constants for hardware
  (e.g. ccr), accelerated software (e.g. aesni), and plain software
  (cryptosoft) that give preference in that order.  One effect of this
  is that if you request only hardware when creating a new session,
  you will no longer get a session using accelerated software.
  Another effect is that the default setting to disallow software
  crypto via /dev/crypto now disables accelerated software.

  Once a driver is chosen, 'crypto_newsession' is invoked as before.

- Crypto operations are now solely described by the flat 'cryptop'
  structure.  The linked list of descriptors has been removed.

  A separate enum has been added to describe the type of data buffer
  in use instead of using CRYPTO_F_* flags to make it easier to add
  more types in the future if needed (e.g. wired userspace buffers for
  zero-copy).  It will also make it easier to re-introduce separate
  input and output buffers (in-kernel TLS would benefit from this).

  Try to make the flags related to IV handling less insane:

  - CRYPTO_F_IV_SEPARATE means that the IV is stored in the 'crp_iv'
    member of the operation structure.  If this flag is not set, the
    IV is stored in the data buffer at the 'crp_iv_start' offset.

  - CRYPTO_F_IV_GENERATE means that a random IV should be generated
    and stored into the data buffer.  This cannot be used with
    CRYPTO_F_IV_SEPARATE.

  If a consumer wants to deal with explicit vs implicit IVs, etc. it
  can always generate the IV however it needs and store partial IVs in
  the buffer and the full IV/nonce in crp_iv and set
  CRYPTO_F_IV_SEPARATE.

  The layout of the buffer is now described via fields in cryptop.
  crp_aad_start and crp_aad_length define the boundaries of any AAD.
  Previously with GCM and CCM you defined an auth crd with this range,
  but for ETA your auth crd had to span both the AAD and plaintext
  (and they had to be adjacent).

  crp_payload_start and crp_payload_length define the boundaries of
  the plaintext/ciphertext.  Modes that only do a single operation
  (COMPRESS, CIPHER, DIGEST) should only use this region and leave the
  AAD region empty.

  If a digest is present (or should be generated), it's starting
  location is marked by crp_digest_start.

  Instead of using the CRD_F_ENCRYPT flag to determine the direction
  of the operation, cryptop now includes an 'op' field defining the
  operation to perform.  For digests I've added a new VERIFY digest
  mode which assumes a digest is present in the input and fails the
  request with EBADMSG if it doesn't match the internally-computed
  digest.  GCM and CCM already assumed this, and the new AEAD mode
  requires this for decryption.  The new ETA mode now also requires
  this for decryption, so IPsec and GELI no longer do their own
  authentication verification.  Simple DIGEST operations can also do
  this, though there are no in-tree consumers.

  To eventually support some refcounting to close races, the session
  cookie is now passed to crypto_getop() and clients should no longer
  set crp_sesssion directly.

- Assymteric crypto operation structures should be allocated via
  crypto_getkreq() and freed via crypto_freekreq().  This permits the
  crypto layer to track open asym requests and close races with a
  driver trying to unregister while asym requests are in flight.

- crypto_copyback, crypto_copydata, crypto_apply, and
  crypto_contiguous_subsegment now accept the 'crp' object as the
  first parameter instead of individual members.  This makes it easier
  to deal with different buffer types in the future as well as
  separate input and output buffers.  It's also simpler for driver
  writers to use.

- bus_dmamap_load_crp() loads a DMA mapping for a crypto buffer.
  This understands the various types of buffers so that drivers that
  use DMA do not have to be aware of different buffer types.

- Helper routines now exist to build an auth context for HMAC IPAD
  and OPAD.  This reduces some duplicated work among drivers.

- Key buffers are now treated as const throughout the framework and in
  device drivers.  However, session key buffers provided when a session
  is created are expected to remain alive for the duration of the
  session.

- GCM and CCM sessions now only specify a cipher algorithm and a cipher
  key.  The redundant auth information is not needed or used.

- For cryptosoft, split up the code a bit such that the 'process'
  callback now invokes a function pointer in the session.  This
  function pointer is set based on the mode (in effect) though it
  simplifies a few edge cases that would otherwise be in the switch in
  'process'.

  It does split up GCM vs CCM which I think is more readable even if there
  is some duplication.

- I changed /dev/crypto to support GMAC requests using CRYPTO_AES_NIST_GMAC
  as an auth algorithm and updated cryptocheck to work with it.

- Combined cipher and auth sessions via /dev/crypto now always use ETA
  mode.  The COP_F_CIPHER_FIRST flag is now a no-op that is ignored.
  This was actually documented as being true in crypto(4) before, but
  the code had not implemented this before I added the CIPHER_FIRST
  flag.

- I have not yet updated /dev/crypto to be aware of explicit modes for
  sessions.  I will probably do that at some point in the future as well
  as teach it about IV/nonce and tag lengths for AEAD so we can support
  all of the NIST KAT tests for GCM and CCM.

- I've split up the exising crypto.9 manpage into several pages
  of which many are written from scratch.

- I have converted all drivers and consumers in the tree and verified
  that they compile, but I have not tested all of them.  I have tested
  the following drivers:

  - cryptosoft
  - aesni (AES only)
  - blake2
  - ccr

  and the following consumers:

  - cryptodev
  - IPsec
  - ktls_ocf
  - GELI (lightly)

  I have not tested the following:

  - ccp
  - aesni with sha
  - hifn
  - kgssapi_krb5
  - ubsec
  - padlock
  - safe
  - armv8_crypto (aarch64)
  - glxsb (i386)
  - sec (ppc)
  - cesa (armv7)
  - cryptocteon (mips64)
  - nlmsec (mips64)

Discussed with:	cem
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D23677
2020-03-27 18:25:23 +00:00
kaktus
ad355b0a9d Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)
r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE.  All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by:	kib (mentor, blanket)
Commented by:	kib, gallatin, melifaro
Differential Revision:	https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
emaste
f238fb1582 Miscellaneous typo fixes
Submitted by:	Gordon Bergling <gbergling_gmail.com>
Differential Revision:	https://reviews.freebsd.org/D23453
2020-02-07 19:53:07 +00:00
jhb
982661897c Add stricter checking on mac key lengths.
Negative lengths are always invalid.  The key length should also
be zero for hash algorithms that do not accept a key.

admbugs:	949
Reported by:	Yuval Kanarenstein <yuvalk@ssd-disclosure.com>
Reviewed by:	cem
MFC after:	1 week
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D23094
2020-01-09 18:29:59 +00:00
jhb
be365bf27e Remove no-longer-used function prototype.
Reported by:	amd64-gcc
2020-01-08 22:16:26 +00:00
jhb
1cea1de54f Remove unneeded cdevsw methods and D_NEEDGIANT.
Reviewed by:	kib
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D23079
2020-01-08 19:05:23 +00:00
jhb
dee7ebd77d Use falloc_noinstall + finstall for crypto file descriptors.
Reviewed by:	cem, kib
MFC after:	1 week
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D23078
2020-01-08 19:03:24 +00:00
jhb
f03bc830e7 Add a reference count to cryptodev sessions.
This prevents use-after-free races with crypto requests (which may
sleep) and CIOCFSESSION as well as races from current CIOCFSESSION
requests.

admbugs:	949
Reported by:	Yuval Kanarenstein <yuvalk@ssd-disclosure.com>
Reviewed by:	cem
MFC after:	1 week
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D23077
2020-01-08 18:59:23 +00:00
jhb
6572d0ba22 Add support for TLS 1.3 using AES-GCM to the OCF backend for KTLS.
Reviewed by:	gallatin
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D22802
2019-12-18 01:37:00 +00:00
gallatin
74ca068530 kTLS support for TLS 1.3
TLS 1.3 requires a few changes because 1.3 pretends to be 1.2
with a record type of application data. The "real" record type is
then included at the end of the user-supplied plaintext
data. This required adding a field to the mbuf_ext_pgs struct to
save the record type, and passing the real record type to the
sw_encrypt() ktls backend functions.

Reviewed by:	jhb, hselasky
Sponsored by:	Netflix
Differential Revision:	D21801
2019-09-27 19:17:40 +00:00
jhb
0ac840bfd0 Adjust the deprecated warnings for /dev/crypto to be less noisy.
Warn when actual operations are performed instead of when sessions are
created.  The /dev/crypto engine in OpenSSL 1.0.x tries to create
sessions for all possible algorithms each time it is initialized
resulting in spurious warnings.

Reported by:	Mike Tancsa
MFC after:	3 days
Sponsored by:	Chelsio Communications
2019-08-27 21:29:37 +00:00
jhb
1cf31620c8 Add kernel-side support for in-kernel TLS.
KTLS adds support for in-kernel framing and encryption of Transport
Layer Security (1.0-1.2) data on TCP sockets.  KTLS only supports
offload of TLS for transmitted data.  Key negotation must still be
performed in userland.  Once completed, transmit session keys for a
connection are provided to the kernel via a new TCP_TXTLS_ENABLE
socket option.  All subsequent data transmitted on the socket is
placed into TLS frames and encrypted using the supplied keys.

Any data written to a KTLS-enabled socket via write(2), aio_write(2),
or sendfile(2) is assumed to be application data and is encoded in TLS
frames with an application data type.  Individual records can be sent
with a custom type (e.g. handshake messages) via sendmsg(2) with a new
control message (TLS_SET_RECORD_TYPE) specifying the record type.

At present, rekeying is not supported though the in-kernel framework
should support rekeying.

KTLS makes use of the recently added unmapped mbufs to store TLS
frames in the socket buffer.  Each TLS frame is described by a single
ext_pgs mbuf.  The ext_pgs structure contains the header of the TLS
record (and trailer for encrypted records) as well as references to
the associated TLS session.

KTLS supports two primary methods of encrypting TLS frames: software
TLS and ifnet TLS.

Software TLS marks mbufs holding socket data as not ready via
M_NOTREADY similar to sendfile(2) when TLS framing information is
added to an unmapped mbuf in ktls_frame().  ktls_enqueue() is then
called to schedule TLS frames for encryption.  In the case of
sendfile_iodone() calls ktls_enqueue() instead of pru_ready() leaving
the mbufs marked M_NOTREADY until encryption is completed.  For other
writes (vn_sendfile when pages are available, write(2), etc.), the
PRUS_NOTREADY is set when invoking pru_send() along with invoking
ktls_enqueue().

A pool of worker threads (the "KTLS" kernel process) encrypts TLS
frames queued via ktls_enqueue().  Each TLS frame is temporarily
mapped using the direct map and passed to a software encryption
backend to perform the actual encryption.

(Note: The use of PHYS_TO_DMAP could be replaced with sf_bufs if
someone wished to make this work on architectures without a direct
map.)

KTLS supports pluggable software encryption backends.  Internally,
Netflix uses proprietary pure-software backends.  This commit includes
a simple backend in a new ktls_ocf.ko module that uses the kernel's
OpenCrypto framework to provide AES-GCM encryption of TLS frames.  As
a result, software TLS is now a bit of a misnomer as it can make use
of hardware crypto accelerators.

Once software encryption has finished, the TLS frame mbufs are marked
ready via pru_ready().  At this point, the encrypted data appears as
regular payload to the TCP stack stored in unmapped mbufs.

ifnet TLS permits a NIC to offload the TLS encryption and TCP
segmentation.  In this mode, a new send tag type (IF_SND_TAG_TYPE_TLS)
is allocated on the interface a socket is routed over and associated
with a TLS session.  TLS records for a TLS session using ifnet TLS are
not marked M_NOTREADY but are passed down the stack unencrypted.  The
ip_output_send() and ip6_output_send() helper functions that apply
send tags to outbound IP packets verify that the send tag of the TLS
record matches the outbound interface.  If so, the packet is tagged
with the TLS send tag and sent to the interface.  The NIC device
driver must recognize packets with the TLS send tag and schedule them
for TLS encryption and TCP segmentation.  If the the outbound
interface does not match the interface in the TLS send tag, the packet
is dropped.  In addition, a task is scheduled to refresh the TLS send
tag for the TLS session.  If a new TLS send tag cannot be allocated,
the connection is dropped.  If a new TLS send tag is allocated,
however, subsequent packets will be tagged with the correct TLS send
tag.  (This latter case has been tested by configuring both ports of a
Chelsio T6 in a lagg and failing over from one port to another.  As
the connections migrated to the new port, new TLS send tags were
allocated for the new port and connections resumed without being
dropped.)

ifnet TLS can be enabled and disabled on supported network interfaces
via new '[-]txtls[46]' options to ifconfig(8).  ifnet TLS is supported
across both vlan devices and lagg interfaces using failover, lacp with
flowid enabled, or lacp with flowid enabled.

Applications may request the current KTLS mode of a connection via a
new TCP_TXTLS_MODE socket option.  They can also use this socket
option to toggle between software and ifnet TLS modes.

In addition, a testing tool is available in tools/tools/switch_tls.
This is modeled on tcpdrop and uses similar syntax.  However, instead
of dropping connections, -s is used to force KTLS connections to
switch to software TLS and -i is used to switch to ifnet TLS.

Various sysctls and counters are available under the kern.ipc.tls
sysctl node.  The kern.ipc.tls.enable node must be set to true to
enable KTLS (it is off by default).  The use of unmapped mbufs must
also be enabled via kern.ipc.mb_use_ext_pgs to enable KTLS.

KTLS is enabled via the KERN_TLS kernel option.

This patch is the culmination of years of work by several folks
including Scott Long and Randall Stewart for the original design and
implementation; Drew Gallatin for several optimizations including the
use of ext_pgs mbufs, the M_NOTREADY mechanism for TLS records
awaiting software encryption, and pluggable software crypto backends;
and John Baldwin for modifications to support hardware TLS offload.

Reviewed by:	gallatin, hselasky, rrs
Obtained from:	Netflix
Sponsored by:	Netflix, Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21277
2019-08-27 00:01:56 +00:00
jhb
abcc5624ea Stop using des_cblock * for arguments to DES functions.
This amounts to a char ** since it is a char[8] *.  Evil casts mostly
resolved the fact that what was actually passed in were plain char *.
Instead, change the DES functions to use 'unsigned char *' for keys
and for input and output buffers.

Reviewed by:	cem, imp
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21389
2019-08-26 17:25:07 +00:00
jhb
f2015b4065 Use 'const' for keys and IVs passed to software encryption algorithms.
Specifically, use 'const' for the key passed to the 'setkey' method
and 'const' for the 'iv' passed to the 'reinit' method.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21347
2019-08-22 00:02:08 +00:00
delphij
83948bcf4b Allow Kernel to link in both legacy libkern/zlib and new sys/contrib/zlib,
with an eventual goal to convert all legacl zlib callers to the new zlib
version:

 * Move generic zlib shims that are not specific to zlib 1.0.4 to
   sys/dev/zlib.
 * Connect new zlib (1.2.11) to the zlib kernel module, currently built
   with Z_SOLO.
 * Prefix the legacy zlib (1.0.4) with 'zlib104_' namespace.
 * Convert sys/opencrypto/cryptodeflate.c to use new zlib.
 * Remove bundled zlib 1.2.3 from ZFS and adapt it to new zlib and make
   it depend on the zlib module.
 * Fix Z_SOLO build of new zlib.

PR:		229763
Submitted by:	Yoshihiro Ota <ota j email ne jp>
Reviewed by:	markm (sys/dev/zlib/zlib_kmod.c)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D19706
2019-08-01 06:35:33 +00:00
jhb
562c25dfaf Move declaration of warninterval out from under COMPAT_FREEBSD32.
This fixes builds of kernels without COMPAT_FREEBSD32.

Reported by:	tinderbox
MFC after:	1 month
2019-06-11 23:28:07 +00:00
jhb
7f6f4a4c07 Make the warning intervals for deprecated crypto algorithms tunable.
New sysctl/tunables can now set the interval (in seconds) between
rate-limited crypto warnings.  The new sysctls are:
- kern.cryptodev_warn_interval for /dev/crypto
- net.inet.ipsec.crypto_warn_interval for IPsec
- kern.kgssapi_warn_interval for KGSSAPI

Reviewed by:	cem
MFC after:	1 month
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20555
2019-06-11 23:00:55 +00:00
jhb
ea1c54fcf5 Add warnings to /dev/crypto for deprecated algorithms.
These algorithms are deprecated algorithms that will have no in-kernel
consumers in FreeBSD 13.  Specifically, deprecate the following
algorithms:
- ARC4
- Blowfish
- CAST128
- DES
- 3DES
- MD5-HMAC
- Skipjack

MFC after:	1 month
Relnotes:	yes
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20554
2019-06-10 19:26:57 +00:00
delphij
f389e8a103 cryptodeflate: Drop z_stream zbuf.state->dummy from SDT probe.
For older versions of zlib, dummy was a workaround for compilers that do not
handle opaque type definition well; on FreeBSD, it's representing a value
that is not really useful for monitoring purposes, and the field would be gone
in newer zlib versions.

PR:		229763
Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
Differential Revision:	https://reviews.freebsd.org/D20222
2019-05-24 02:44:15 +00:00
jhb
e8c3f745e3 Don't panic for empty CCM requests.
A request to encrypt an empty payload without any AAD is unusual, but
it is defined behavior.  Removing this assertion removes a panic and
instead returns the correct tag for an empty buffer.

Reviewed by:	cem, sef
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20043
2019-04-24 23:27:39 +00:00
sef
7f9195f715 Fix another bug introduced during the review process of r344140:
the tag wasn't being computed properly due to chaning a >= comparison
to an == comparison.

Specifically:  CBC-MAC encodes the length of the authorization data
into the the stream to be encrypted/hashed.  For short data, this is
two bytes (big-endian 16 bit value); for larger data, it's 6 bytes
(a prefix of 0xff, 0xfe, followed by a 32-bit big-endian length).  And
there's a larger size, which is 10 bytes.  These extra bytes weren't
being accounted for with the post-review code.  The other bit that then came
into play was that OCF only calls the Update code with blksiz=16, which
meant that I had to ignore the length variable.  (It also means that it
can't be called with a single buffer containing the AAD and payload;
however, OCF doesn't do this for the software-only algorithsm.)

I tested with this script:

ALG=aes-ccm
DEV=soft

for aad in 0 1 2 3 4 14 16 24 30 32 34 36 1020
do
        for dln in 16 32 1024 2048 10240
        do
                echo "Testing AAD length ${aad} data length ${dln}"
                /root/cryptocheck -A ${aad} -a ${ALG} -d ${DEV} ${dln}
        done
done

Reviewed by:	cem
Sponsored by:	iXsystems Inc.
2019-02-25 19:14:16 +00:00
sef
c9afb7f6b3 Fix another issue from r344141, having to do with size of a shift amount.
This did not show up in my testing.

Differential Revision:	https://reviews.freebsd.org/D18592
2019-02-15 04:15:43 +00:00
sef
ed05b7c613 Pasting in a source control line missed the last quote. Fixed. 2019-02-15 04:01:59 +00:00
sef
1c55bdab4d Add AES-CCM encryption, and plumb into OCF.
This commit essentially has three parts:

* Add the AES-CCM encryption hooks.  This is in and of itself fairly small,
as there is only a small difference between CCM and the other ICM-based
algorithms.
* Hook the code into the OpenCrypto framework.  This is the bulk of the
changes, as the algorithm type has to be checked for, and the differences
between it and GCM dealt with.
* Update the cryptocheck tool to be aware of it.  This is invaluable for
confirming that the code works.

This is a software-only implementation, meaning that the performance is very
low.

Sponsored by:	iXsystems Inc.
Differential Revision:	https://reviews.freebsd.org/D19090
2019-02-15 03:53:03 +00:00
sef
e1cdc83e7d Add CBC-MAC authentication.
This adds the CBC-MAC code to the kernel, but does not hook it up to
anything (that comes in the next commit).

https://tools.ietf.org/html/rfc3610 describes the algorithm.

Note that this is a software-only implementation, which means it is
fairly slow.

Sponsored by:   iXsystems Inc
Differential Revision:  https://reviews.freebsd.org/D18592
2019-02-15 03:46:39 +00:00
marius
dc4502cae7 As struct cryptop is wrapped in #ifdef _KERNEL, userland doesn't
need to drag in <sys/_task.h> either.
2019-02-10 21:27:03 +00:00