Commit Graph

159 Commits

Author SHA1 Message Date
John Baldwin
5425750f03 Move per-operation data out of the csession structure.
Create a struct cryptop_data which contains state needed for a single
symmetric crypto operation and move that state out of the session. This
closes a race with the CRYPTO_F_DONE flag that can result in use after
free.

While here, remove the 'cse->error' member.  It was just a copy of
'crp->crp_etype' and cryptodev_op() and cryptodev_aead() checked both
'crp->crp_etype' and 'cse->error'.  Similarly, do not check for an
error from mtx_sleep() since it is not used with PCATCH or a timeout
so cannot fail with an error.

PR:		218597
Reviewed by:	kib
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13928
2018-01-26 23:21:50 +00:00
John Baldwin
a296c71966 Split crp_buf into a union.
This adds explicit crp_mbuf and crp_uio pointers of the right type to
replace casts of crp_buf.  This does not sweep through changing existing
code, but new code should use the correct fields instead of casts.

Reviewed by:	kib
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13927
2018-01-16 19:41:18 +00:00
John Baldwin
eda08c83c7 Change the type of 'crp_opaque' from caddr_t to void *.
Opaque pointers should be void *.  Note that this does not go through
the tree removing all of the now-unnecessary casts.

Reviewed by:	kib
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13848
2018-01-11 18:09:24 +00:00
John Baldwin
eb22dfe39b Axe tmp_iv from the cryptodev session structure.
Just copyin the IV into the crypto descriptor directly.  This avoids
copying the IV twice for each operation.

Reviewed by:	kib
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D13847
2018-01-11 18:07:21 +00:00
John Baldwin
776a2127ef Flesh out static dtrace probes for /dev/crypto ioctl errors.
In particular, no probes were present for AEAD requests, but also for
some other error cases in other ioctl requests.

MFC after:	2 weeks
Sponsored by:	Chelsio Communications
2018-01-11 00:22:24 +00:00
Fabien Thomas
de2b2c908a Fix uninitialized crp_retw_id when using asynchronous crypto drivers
with defered callbacks.

Submitted by:	emeric.poupon@stormshield.eu
Reported by:	mav@
Reviewed by:	fabient@
2018-01-08 13:43:12 +00:00
John Baldwin
8bbeea2b1d Remove a redunant check. 2017-12-30 03:08:49 +00:00
Alexander Kabaev
151ba7933a Do pass removing some write-only variables from the kernel.
This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385
2017-12-25 04:48:39 +00:00
Warner Losh
3a7d67e741 We don't need both _STAND and _STANDALONE. There's more places that
use _STANDALONE, so change the former to the latter.

Sponsored by: Netflix
2017-12-02 00:07:09 +00:00
Fabien Thomas
39bbca6ffd crypto(9) is called from ipsec in CRYPTO_F_CBIFSYNC mode. This is working
fine when a lot of different flows to be ciphered/deciphered are involved.

However, when a software crypto driver is used, there are
situations where we could benefit from making crypto(9) multi threaded:
- a single flow is to be ciphered: only one thread is used to cipher it,
- a single ESP flow is to be deciphered: only one thread is used to
decipher it.

The idea here is to call crypto(9) using a new mode (CRYPTO_F_ASYNC) to
dispatch the crypto jobs on multiple threads, if the underlying crypto
driver is working in synchronous mode.

Another flag is added (CRYPTO_F_ASYNC_KEEPORDER) to make crypto(9)
dispatch the crypto jobs in the order they are received (an additional
queue/thread is used), so that the packets are reinjected in the network
using the same order they were posted.

A new sysctl net.inet.ipsec.async_crypto can be used to activate
this new behavior (disabled by default).

Submitted by:	Emeric Poupon <emeric.poupon@stormshield.eu>
Reviewed by:	ae, jmg, jhb
Differential Revision:    https://reviews.freebsd.org/D10680
Sponsored by:	Stormshield
2017-11-03 10:27:22 +00:00
Conrad Meyer
d7d2f0d4d1 crypto(9): Print flags in more useful hex
Sponsored by:	Dell EMC Isilon
2017-10-11 20:04:30 +00:00
Conrad Meyer
255811d758 opencrypto: Use C99 initializers for auth_hash instances
A misordering in the Via padlock driver really strongly suggested that these
should use C99 named initializers.

No functional change.

Sponsored by:	Dell EMC Isilon
2017-09-26 17:52:52 +00:00
Conrad Meyer
3693b18840 opencrypto: Loosen restriction on HMAC key sizes
Theoretically, HMACs do not actually have any limit on key sizes.
Transforms should compact input keys larger than the HMAC block size by
using the transform (hash) on the input key.

(Short input keys are padded out with zeros to the HMAC block size.)

Still, not all FreeBSD crypto drivers that provide HMAC functionality
handle longer-than-blocksize keys appropriately, so enforce a "maximum" key
length in the crypto API for auth_hashes that previously expressed a
requirement.  (The "maximum" is the size of a single HMAC block for the
given transform.)  Unconstrained auth_hashes are left as-is.

I believe the previous hardcoded sizes were committed in the original
import of opencrypto from OpenBSD and are due to specific protocol
details of IPSec.  Note that none of the previous sizes actually matched
the appropriate HMAC block size.

The previous hardcoded sizes made the SHA tests in cryptotest.py
useless for testing FreeBSD crypto drivers; none of the NIST-KAT example
inputs had keys sized to the previous expectations.

The following drivers were audited to check that they handled keys up to
the block size of the HMAC safely:

  Software HMAC:
    * padlock(4)
    * cesa
    * glxsb
    * safe(4)
    * ubsec(4)

  Hardware accelerated HMAC:
    * ccr(4)
    * hifn(4)
    * sec(4) (Only supports up to 64 byte keys despite claiming to
      support SHA2 HMACs, but validates input key sizes)
    * cryptocteon (MIPS)
    * nlmsec (MIPS)
    * rmisec (MIPS) (Amusingly, does not appear to use key material at
      all -- presumed broken)

Reviewed by:	jhb (previous version), rlibby (previous version)
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12437
2017-09-26 16:18:10 +00:00
Conrad Meyer
a317fb03c2 crypto(9): Use a more specific error code when a capable driver is not found
When crypto_newsession() is given a request for an unsupported capability,
raise a more specific error than EINVAL.

This allows cryptotest.py to skip some HMAC tests that a driver does not
support.

Reviewed by:	jhb, rlibby
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12451
2017-09-26 01:31:49 +00:00
John Baldwin
cc05c7d256 Support AEAD requests with non-GCM algorithms.
In particular, support chaining an AES cipher with an HMAC for a request
including AAD.  This permits submitting requests from userland to encrypt
objects like IPSec packets using these algorithms.

In the non-GCM case, the authentication crypto descriptor covers both the
AAD and the ciphertext.  The GCM case remains unchanged.  This matches
the requests created internally in IPSec.  For the non-GCM case, the
COP_F_CIPHER_FIRST is also supported since the ordering matters.

Note that while this can be used to simulate IPSec requests from userland,
this ioctl cannot currently be used to perform TLS requests using AES-CBC
and MAC-before-encrypt.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D11759
2017-09-22 00:34:46 +00:00
John Baldwin
2c907637bc Add a new COP_F_CIPHER_FIRST flag for struct crypt_op.
This requests that the cipher be performed before rather than after
the HMAC when both are specified for a single operation.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D11757
2017-09-22 00:21:58 +00:00
John Baldwin
95f076384f Place the AAD before the plaintext/ciphertext for CIOCRYPTAEAD.
Software crypto implementations don't care how the buffer is laid out,
but hardware implementations may assume that the AAD is always before
the plain/cipher text and that the hash/tag is immediately after the end
of the plain/cipher text.

In particular, this arrangement matches the layout of both IPSec packets
and TLS frames.  Linux's crypto framework also assumes this layout for
AEAD requests.

Reviewed by:	cem
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D11758
2017-09-22 00:15:54 +00:00
John Baldwin
7dd10fded0 Use const with some read-only buffers in opencrypto APIs.
- Mark the source buffer for a copyback operation as const in the kernel
  API.
- Use const with input-only buffers in crypto ioctl structures used with
  /dev/crypto.

MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D10517
2017-05-09 18:28:42 +00:00
John Baldwin
2e2e26d14e Don't leak a session and lock if a GMAC key has an invalid length.
Reviewed by:	delphij (secteam)
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D10273
2017-04-05 01:46:41 +00:00
Allan Jude
a01b1aa787 remove machine/metadata.h from sys/opencrypto/crypto.c
The header was added by the recent keybuf feature (r316343)

MODINFOMD_KEYBUF originally resided here, but was moved to linker.h

This change fixes the build on risc-5 which doesn't have a metadata.h

Detected by Jenkins: https://ci.freebsd.org/job/FreeBSD-head-riscv64-build/1167/console

Reported by:	lwhsu
2017-04-04 04:01:02 +00:00
Allan Jude
ec5c0e5be9 Implement boot-time encryption key passing (keybuf)
This patch adds a general mechanism for providing encryption keys to the
kernel from the boot loader. This is intended to enable GELI support at
boot time, providing a better mechanism for passing keys to the kernel
than environment variables. It is designed to be extensible to other
applications, and can easily handle multiple encrypted volumes with
different keys.

This mechanism is currently used by the pending GELI EFI work.
Additionally, this mechanism can potentially be used to interface with
GRUB, opening up options for coreboot+GRUB configurations with completely
encrypted disks.

Another benefit over the existing system is that it does not require
re-deriving the user key from the password at each boot stage.

Most of this patch was written by Eric McCorkle. It was extended by
Allan Jude with a number of minor enhancements and extending the keybuf
feature into boot2.

GELI user keys are now derived once, in boot2, then passed to the loader,
which reuses the key, then passes it to the kernel, where the GELI module
destroys the keybuf after decrypting the volumes.

Submitted by:	Eric McCorkle <eric@metricspace.net> (Original Version)
Reviewed by:	oshogbo (earlier version), cem (earlier version)
MFC after:	3 weeks
Relnotes:	yes
Sponsored by:	ScaleEngine Inc.
Differential Revision:	https://reviews.freebsd.org/D9575
2017-04-01 05:05:22 +00:00
Andrew Turner
6ed982a221 Add support for the fpu_kern(9) KPI on arm64. It hooks into the existing
VFP code to store the old context, with lazy loading of the new context
when needed.

FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.

Reviewed by:	kib
Obtained from:	ABT Systeems Ltd
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D8276
2016-10-20 09:22:10 +00:00
Conrad Meyer
bf4c239e47 opencrypto AES-ICM: Fix heap corruption typo
This error looks like it was a simple copy-paste typo in the original commit
for this code (r275732).

PR:		204009
Reported by:	Chang-Hsien Tsai <luke.tw AT gmail.com>
Sponsored by:	EMC / Isilon Storage Division
2016-08-01 22:57:03 +00:00
Pedro F. Giffuni
1762773d79 sys/opencrypto: minor spelling fixes.
No functional change.

Reviewed by:	jmg
2016-05-06 23:37:19 +00:00
Allan Jude
2155bb238f Break up opencrypto/xform.c so it can be reused piecemeal
Keep xform.c as a meta-file including the broken out bits
existing code that includes xform.c continues to work as normal

Individual algorithms can now be reused elsewhere, including outside
of the kernel

Reviewed by:	bapt (previous version), gnn, delphij
Approved by:	secteam
MFC after:	1 week
Sponsored by:	ScaleEngine Inc.
Differential Revision:	https://reviews.freebsd.org/D4674
2015-12-30 22:43:07 +00:00
Allan Jude
7a3f5d11fb Replace sys/crypto/sha2/sha2.c with lib/libmd/sha512c.c
cperciva's libmd implementation is 5-30% faster

The same was done for SHA256 previously in r263218

cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation

Extend sbin/md5 to create sha384(1)

Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}

Reviewed by:	cperciva, des, delphij
Approved by:	secteam, bapt (mentor)
MFC after:	2 weeks
Sponsored by:	ScaleEngine Inc.
Differential Revision:	https://reviews.freebsd.org/D3929
2015-12-27 17:33:59 +00:00
Mark Johnston
7672ca059a Remove unneeded includes of opt_kdtrace.h.
As of r258541, KDTRACE_HOOKS is defined in opt_global.h, so opt_kdtrace.h
is not needed when defining SDT(9) probes.
2015-11-22 02:01:01 +00:00
Andrey V. Elsukov
0c80e7df43 Use explicitly specified ivsize instead of blocksize when we mean IV size.
Set zero ivsize for enc_xform_null and remove special handling from
xform_esp.c.

Reviewed by:	gnn
Differential Revision:	https://reviews.freebsd.org/D1503
2015-11-16 07:10:42 +00:00
John-Mark Gurney
a2bc81bf7c Make IPsec work with AES-GCM and AES-ICM (aka CTR) in OCF... IPsec
defines the keys differently than NIST does, so we have to muck with
key lengths and nonce/IVs to be standard compliant...

Remove the iv from secasvar as it was unused...

Add a counter protected by a mutex to ensure that the counter for GCM
and ICM will never be repeated..  This is a requirement for security..
I would use atomics, but we don't have a 64bit one on all platforms..

Fix a bug where IPsec was depending upon the OCF to ensure that the
blocksize was always at least 4 bytes to maintain alignment... Move
this logic into IPsec so changes to OCF won't break IPsec...

In one place, espx was always non-NULL, so don't test that it's
non-NULL before doing work..

minor style cleanups...

drop setting key and klen as they were not used...

Enforce that OCF won't pass invalid key lengths to AES that would
panic the machine...

This was has been tested by others too...  I tested this against
NetBSD 6.1.5 using mini-test suite in
https://github.com/jmgurney/ipseccfgs and the only things that don't
pass are keyed md5 and sha1, and 3des-deriv (setkey syntax error),
all other modes listed in setkey's man page...  The nice thing is
that NetBSD uses setkey, so same config files were used on both...

Reviewed by:	gnn
2015-08-04 17:47:11 +00:00
John-Mark Gurney
577f7474b0 Fix XTS, and name things a bit better...
Though confusing, GCM using ICM_BLOCK_LEN, but ICM does not is
correct...  GCM is built on ICM, but uses a function other than
swcr_encdec...  swcr_encdec cannot handle partial blocks which is
why it must still use AES_BLOCK_LEN and is why XTS was broken by the
commit...

Thanks to the tests for helping sure I didn't break GCM w/ an earlier
patch...

I did run the tests w/o this patch, and need to figure out why they
did not fail, clearly more tests are needed...

Prodded by:	peter
2015-07-14 07:45:18 +00:00
George V. Neville-Neil
16de9ac1b5 Add support for AES modes to IPSec. These modes work both in software only
mode and with hardware support on systems that have AESNI instructions.

Differential Revision:	D2936
Reviewed by:	jmg, eri, cognet
Sponsored by:	Rubicon Communications (Netgate)
2015-07-09 18:16:35 +00:00
John-Mark Gurney
748a12e2c3 we may get here w/ non-sleepable locks held, so switch to _NOWAIT when
doing this memory allocation...

Reviewed by:	ae
2015-07-07 18:45:32 +00:00
John-Mark Gurney
fb17b4c55c change the KASSERT so it is meaningful, var is unsigned, so even
when it wraps, it's still >= 0...

Reported by:	Coverity
CID:		1017564
2015-06-19 22:42:26 +00:00
Mark Johnston
d0f6fcd886 Use the correct number of arguments for the
opencrypto:deflate:deflate_global:bad DTrace probe, which is defined to
have

MFC after:	1 week
2015-05-25 01:31:39 +00:00
Craig Rodrigues
d9db52256e Move zlib.c from net to libkern.
It is not network-specific code and would
be better as part of libkern instead.
Move zlib.h and zutil.h from net/ to sys/
Update includes to use sys/zlib.h and sys/zutil.h instead of net/

Submitted by:		Steve Kiernan stevek@juniper.net
Obtained from:		Juniper Networks, Inc.
GitHub Pull Request:	https://github.com/freebsd/freebsd/pull/28
Relnotes:		yes
2015-04-22 14:38:58 +00:00
John-Mark Gurney
9df9c6bc0d add the define to properly guard this header..
Sponsored by:	Netflix, Inc.
2015-04-07 09:00:03 +00:00
John-Mark Gurney
08fca7a56b Add some new modes to OpenCrypto. These modes are AES-ICM (can be used
for counter mode), and AES-GCM.  Both of these modes have been added to
the aesni module.

Included is a set of tests to validate that the software and aesni
module calculate the correct values.  These use the NIST KAT test
vectors.  To run the test, you will need to install a soon to be
committed port, nist-kat that will install the vectors.  Using a port
is necessary as the test vectors are around 25MB.

All the man pages were updated.  I have added a new man page, crypto.7,
which includes a description of how to use each mode.  All the new modes
and some other AES modes are present.  It would be good for someone
else to go through and document the other modes.

A new ioctl was added to support AEAD modes which AES-GCM is one of them.
Without this ioctl, it is not possible to test AEAD modes from userland.

Add a timing safe bcmp for use to compare MACs.  Previously we were using
bcmp which could leak timing info and result in the ability to forge
messages.

Add a minor optimization to the aesni module so that single segment
mbufs don't get copied and instead are updated in place.  The aesni
module needs to be updated to support blocked IO so segmented mbufs
don't have to be copied.

We require that the IV be specified for all calls for both GCM and ICM.
This is to ensure proper use of these functions.

Obtained from:	p4: //depot/projects/opencrypto
Relnotes:	yes
Sponsored by:	FreeBSD Foundation
Sponsored by:	NetGate
2014-12-12 19:56:36 +00:00
John Baldwin
cd5d81ba41 Fix build for kernels without COMPAT_FREEBSD32. 2014-09-22 17:32:27 +00:00
John Baldwin
9696feebe2 Add a new fo_fill_kinfo fileops method to add type-specific information to
struct kinfo_file.
- Move the various fill_*_info() methods out of kern_descrip.c and into the
  various file type implementations.
- Rework the support for kinfo_ofile to generate a suitable kinfo_file object
  for each file and then convert that to a kinfo_ofile structure rather than
  keeping a second, different set of code that directly manipulates
  type-specific file information.
- Remove the shm_path() and ksem_info() layering violations.

Differential Revision:	https://reviews.freebsd.org/D775
Reviewed by:	kib, glebius (earlier version)
2014-09-22 16:20:47 +00:00
John Baldwin
2d69d0dcc2 Fix various issues with invalid file operations:
- Add invfo_rdwr() (for read and write), invfo_ioctl(), invfo_poll(),
  and invfo_kqfilter() for use by file types that do not support the
  respective operations.  Home-grown versions of invfo_poll() were
  universally broken (they returned an errno value, invfo_poll()
  uses poll_no_poll() to return an appropriate event mask).  Home-grown
  ioctl routines also tended to return an incorrect errno (invfo_ioctl
  returns ENOTTY).
- Use the invfo_*() functions instead of local versions for
  unsupported file operations.
- Reorder fileops members to match the order in the structure definition
  to make it easier to spot missing members.
- Add several missing methods to linuxfileops used by the OFED shim
  layer: fo_write(), fo_truncate(), fo_kqfilter(), and fo_stat().  Most
  of these used invfo_*(), but a dummy fo_stat() implementation was
  added.
2014-09-12 21:29:10 +00:00
John-Mark Gurney
6c20d7a3ce allow the kern.cryptodevallowsoft sysctl to enable symetric/hashes too...
This will allow us to more easily test the software versions of these
routines...

Considering that we've never had an software asymetric implmentation,
it's doubtful anyone has this enabled...
2014-03-11 01:45:46 +00:00
John-Mark Gurney
1bf557366c some minor clean up.. Always _ZERO memory so mtx_init won't panic...
use the proper macro instead of hand rolling it...

Reviewed by:	jhb (only the malloc change)
MFC after:	1 week
2014-03-11 01:41:09 +00:00
Benno Rice
109919c67a Prevent races in accesses of the software crypto session array.
swcr_newsession can change the pointer for swcr_sessions which races with
swcr_process which is looking up entries in this array.

Add a rwlock that protects changes to the array pointer so that
swcr_newsession and swcr_process no longer race.

Original patch by:	Steve O'Hara-Smith <Steve.OHaraSmith@isilon.com>
Reviewed by:		jmg
Sponsored by:		EMC / Isilon Storage Division
2014-01-28 22:02:29 +00:00
Peter Wemm
f5fdfd5db7 Fix build with 'crypto' and 'zfs' in the same kernel. This was the path
of least pain I could find.
2013-12-08 21:59:46 +00:00
Andriy Gapon
d9fae5ab88 dtrace sdt: remove the ugly sname parameter of SDT_PROBE_DEFINE
In its stead use the Solaris / illumos approach of emulating '-' (dash)
in probe names with '__' (two consecutive underscores).

Reviewed by:	markj
MFC after:	3 weeks
2013-11-26 08:46:27 +00:00
Attilio Rao
54366c0bd7 - For kernel compiled only with KDTRACE_HOOKS and not any lock debugging
option, unbreak the lock tracing release semantic by embedding
  calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined
  version of the releasing functions for mutex, rwlock and sxlock.
  Failing to do so skips the lockstat_probe_func invokation for
  unlocking.
- As part of the LOCKSTAT support is inlined in mutex operation, for
  kernel compiled without lock debugging options, potentially every
  consumer must be compiled including opt_kdtrace.h.
  Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the
  dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES
  is linked there and it is only used as a compile-time stub [0].

[0] immediately shows some new bug as DTRACE-derived support for debug
in sfxge is broken and it was never really tested.  As it was not
including correctly opt_kdtrace.h before it was never enabled so it
was kept broken for a while.  Fix this by using a protection stub,
leaving sfxge driver authors the responsibility for fixing it
appropriately [1].

Sponsored by:	EMC / Isilon storage division
Discussed with:	rstone
[0] Reported by:	rstone
[1] Discussed with:	philip
2013-11-25 07:38:45 +00:00
Gleb Smirnoff
ca04d21d5f Make sendfile() a method in the struct fileops. Currently only
vnode backed file descriptors have this method implemented.

Reviewed by:	kib
Sponsored by:	Nginx, Inc.
Sponsored by:	Netflix
2013-08-15 07:54:31 +00:00
Pawel Jakub Dawidek
45b56a6ba2 When porting XTS-related code from OpenBSD I forgot to update copyright (only
OpenBSD was credited in one of two commits). Fix it.

Reported by:	Theo de Raadt <deraadt@cvs.openbsd.org>
Reviewed by:	Damien Miller <djm@mindrot.org>
2013-02-20 22:59:53 +00:00
Kevin Lo
9823d52705 Revert previous commit...
Pointyhat to:	kevlo (myself)
2012-10-10 08:36:38 +00:00
Kevin Lo
a10cee30c9 Prefer NULL over 0 for pointers 2012-10-09 08:27:40 +00:00