This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.
Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385
fine when a lot of different flows to be ciphered/deciphered are involved.
However, when a software crypto driver is used, there are
situations where we could benefit from making crypto(9) multi threaded:
- a single flow is to be ciphered: only one thread is used to cipher it,
- a single ESP flow is to be deciphered: only one thread is used to
decipher it.
The idea here is to call crypto(9) using a new mode (CRYPTO_F_ASYNC) to
dispatch the crypto jobs on multiple threads, if the underlying crypto
driver is working in synchronous mode.
Another flag is added (CRYPTO_F_ASYNC_KEEPORDER) to make crypto(9)
dispatch the crypto jobs in the order they are received (an additional
queue/thread is used), so that the packets are reinjected in the network
using the same order they were posted.
A new sysctl net.inet.ipsec.async_crypto can be used to activate
this new behavior (disabled by default).
Submitted by: Emeric Poupon <emeric.poupon@stormshield.eu>
Reviewed by: ae, jmg, jhb
Differential Revision: https://reviews.freebsd.org/D10680
Sponsored by: Stormshield
A misordering in the Via padlock driver really strongly suggested that these
should use C99 named initializers.
No functional change.
Sponsored by: Dell EMC Isilon
Theoretically, HMACs do not actually have any limit on key sizes.
Transforms should compact input keys larger than the HMAC block size by
using the transform (hash) on the input key.
(Short input keys are padded out with zeros to the HMAC block size.)
Still, not all FreeBSD crypto drivers that provide HMAC functionality
handle longer-than-blocksize keys appropriately, so enforce a "maximum" key
length in the crypto API for auth_hashes that previously expressed a
requirement. (The "maximum" is the size of a single HMAC block for the
given transform.) Unconstrained auth_hashes are left as-is.
I believe the previous hardcoded sizes were committed in the original
import of opencrypto from OpenBSD and are due to specific protocol
details of IPSec. Note that none of the previous sizes actually matched
the appropriate HMAC block size.
The previous hardcoded sizes made the SHA tests in cryptotest.py
useless for testing FreeBSD crypto drivers; none of the NIST-KAT example
inputs had keys sized to the previous expectations.
The following drivers were audited to check that they handled keys up to
the block size of the HMAC safely:
Software HMAC:
* padlock(4)
* cesa
* glxsb
* safe(4)
* ubsec(4)
Hardware accelerated HMAC:
* ccr(4)
* hifn(4)
* sec(4) (Only supports up to 64 byte keys despite claiming to
support SHA2 HMACs, but validates input key sizes)
* cryptocteon (MIPS)
* nlmsec (MIPS)
* rmisec (MIPS) (Amusingly, does not appear to use key material at
all -- presumed broken)
Reviewed by: jhb (previous version), rlibby (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12437
When crypto_newsession() is given a request for an unsupported capability,
raise a more specific error than EINVAL.
This allows cryptotest.py to skip some HMAC tests that a driver does not
support.
Reviewed by: jhb, rlibby
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12451
In particular, support chaining an AES cipher with an HMAC for a request
including AAD. This permits submitting requests from userland to encrypt
objects like IPSec packets using these algorithms.
In the non-GCM case, the authentication crypto descriptor covers both the
AAD and the ciphertext. The GCM case remains unchanged. This matches
the requests created internally in IPSec. For the non-GCM case, the
COP_F_CIPHER_FIRST is also supported since the ordering matters.
Note that while this can be used to simulate IPSec requests from userland,
this ioctl cannot currently be used to perform TLS requests using AES-CBC
and MAC-before-encrypt.
Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11759
This requests that the cipher be performed before rather than after
the HMAC when both are specified for a single operation.
Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11757
Software crypto implementations don't care how the buffer is laid out,
but hardware implementations may assume that the AAD is always before
the plain/cipher text and that the hash/tag is immediately after the end
of the plain/cipher text.
In particular, this arrangement matches the layout of both IPSec packets
and TLS frames. Linux's crypto framework also assumes this layout for
AEAD requests.
Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11758
- Mark the source buffer for a copyback operation as const in the kernel
API.
- Use const with input-only buffers in crypto ioctl structures used with
/dev/crypto.
MFC after: 1 month
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D10517
The header was added by the recent keybuf feature (r316343)
MODINFOMD_KEYBUF originally resided here, but was moved to linker.h
This change fixes the build on risc-5 which doesn't have a metadata.h
Detected by Jenkins: https://ci.freebsd.org/job/FreeBSD-head-riscv64-build/1167/console
Reported by: lwhsu
This patch adds a general mechanism for providing encryption keys to the
kernel from the boot loader. This is intended to enable GELI support at
boot time, providing a better mechanism for passing keys to the kernel
than environment variables. It is designed to be extensible to other
applications, and can easily handle multiple encrypted volumes with
different keys.
This mechanism is currently used by the pending GELI EFI work.
Additionally, this mechanism can potentially be used to interface with
GRUB, opening up options for coreboot+GRUB configurations with completely
encrypted disks.
Another benefit over the existing system is that it does not require
re-deriving the user key from the password at each boot stage.
Most of this patch was written by Eric McCorkle. It was extended by
Allan Jude with a number of minor enhancements and extending the keybuf
feature into boot2.
GELI user keys are now derived once, in boot2, then passed to the loader,
which reuses the key, then passes it to the kernel, where the GELI module
destroys the keybuf after decrypting the volumes.
Submitted by: Eric McCorkle <eric@metricspace.net> (Original Version)
Reviewed by: oshogbo (earlier version), cem (earlier version)
MFC after: 3 weeks
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D9575
VFP code to store the old context, with lazy loading of the new context
when needed.
FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.
Reviewed by: kib
Obtained from: ABT Systeems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8276
This error looks like it was a simple copy-paste typo in the original commit
for this code (r275732).
PR: 204009
Reported by: Chang-Hsien Tsai <luke.tw AT gmail.com>
Sponsored by: EMC / Isilon Storage Division
Keep xform.c as a meta-file including the broken out bits
existing code that includes xform.c continues to work as normal
Individual algorithms can now be reused elsewhere, including outside
of the kernel
Reviewed by: bapt (previous version), gnn, delphij
Approved by: secteam
MFC after: 1 week
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D4674
cperciva's libmd implementation is 5-30% faster
The same was done for SHA256 previously in r263218
cperciva's implementation was lacking SHA-384 which I implemented, validated against OpenSSL and the NIST documentation
Extend sbin/md5 to create sha384(1)
Chase dependancies on sys/crypto/sha2/sha2.{c,h} and replace them with sha512{c.c,.h}
Reviewed by: cperciva, des, delphij
Approved by: secteam, bapt (mentor)
MFC after: 2 weeks
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3929
Set zero ivsize for enc_xform_null and remove special handling from
xform_esp.c.
Reviewed by: gnn
Differential Revision: https://reviews.freebsd.org/D1503
defines the keys differently than NIST does, so we have to muck with
key lengths and nonce/IVs to be standard compliant...
Remove the iv from secasvar as it was unused...
Add a counter protected by a mutex to ensure that the counter for GCM
and ICM will never be repeated.. This is a requirement for security..
I would use atomics, but we don't have a 64bit one on all platforms..
Fix a bug where IPsec was depending upon the OCF to ensure that the
blocksize was always at least 4 bytes to maintain alignment... Move
this logic into IPsec so changes to OCF won't break IPsec...
In one place, espx was always non-NULL, so don't test that it's
non-NULL before doing work..
minor style cleanups...
drop setting key and klen as they were not used...
Enforce that OCF won't pass invalid key lengths to AES that would
panic the machine...
This was has been tested by others too... I tested this against
NetBSD 6.1.5 using mini-test suite in
https://github.com/jmgurney/ipseccfgs and the only things that don't
pass are keyed md5 and sha1, and 3des-deriv (setkey syntax error),
all other modes listed in setkey's man page... The nice thing is
that NetBSD uses setkey, so same config files were used on both...
Reviewed by: gnn
Though confusing, GCM using ICM_BLOCK_LEN, but ICM does not is
correct... GCM is built on ICM, but uses a function other than
swcr_encdec... swcr_encdec cannot handle partial blocks which is
why it must still use AES_BLOCK_LEN and is why XTS was broken by the
commit...
Thanks to the tests for helping sure I didn't break GCM w/ an earlier
patch...
I did run the tests w/o this patch, and need to figure out why they
did not fail, clearly more tests are needed...
Prodded by: peter
mode and with hardware support on systems that have AESNI instructions.
Differential Revision: D2936
Reviewed by: jmg, eri, cognet
Sponsored by: Rubicon Communications (Netgate)
It is not network-specific code and would
be better as part of libkern instead.
Move zlib.h and zutil.h from net/ to sys/
Update includes to use sys/zlib.h and sys/zutil.h instead of net/
Submitted by: Steve Kiernan stevek@juniper.net
Obtained from: Juniper Networks, Inc.
GitHub Pull Request: https://github.com/freebsd/freebsd/pull/28
Relnotes: yes
for counter mode), and AES-GCM. Both of these modes have been added to
the aesni module.
Included is a set of tests to validate that the software and aesni
module calculate the correct values. These use the NIST KAT test
vectors. To run the test, you will need to install a soon to be
committed port, nist-kat that will install the vectors. Using a port
is necessary as the test vectors are around 25MB.
All the man pages were updated. I have added a new man page, crypto.7,
which includes a description of how to use each mode. All the new modes
and some other AES modes are present. It would be good for someone
else to go through and document the other modes.
A new ioctl was added to support AEAD modes which AES-GCM is one of them.
Without this ioctl, it is not possible to test AEAD modes from userland.
Add a timing safe bcmp for use to compare MACs. Previously we were using
bcmp which could leak timing info and result in the ability to forge
messages.
Add a minor optimization to the aesni module so that single segment
mbufs don't get copied and instead are updated in place. The aesni
module needs to be updated to support blocked IO so segmented mbufs
don't have to be copied.
We require that the IV be specified for all calls for both GCM and ICM.
This is to ensure proper use of these functions.
Obtained from: p4: //depot/projects/opencrypto
Relnotes: yes
Sponsored by: FreeBSD Foundation
Sponsored by: NetGate
struct kinfo_file.
- Move the various fill_*_info() methods out of kern_descrip.c and into the
various file type implementations.
- Rework the support for kinfo_ofile to generate a suitable kinfo_file object
for each file and then convert that to a kinfo_ofile structure rather than
keeping a second, different set of code that directly manipulates
type-specific file information.
- Remove the shm_path() and ksem_info() layering violations.
Differential Revision: https://reviews.freebsd.org/D775
Reviewed by: kib, glebius (earlier version)
- Add invfo_rdwr() (for read and write), invfo_ioctl(), invfo_poll(),
and invfo_kqfilter() for use by file types that do not support the
respective operations. Home-grown versions of invfo_poll() were
universally broken (they returned an errno value, invfo_poll()
uses poll_no_poll() to return an appropriate event mask). Home-grown
ioctl routines also tended to return an incorrect errno (invfo_ioctl
returns ENOTTY).
- Use the invfo_*() functions instead of local versions for
unsupported file operations.
- Reorder fileops members to match the order in the structure definition
to make it easier to spot missing members.
- Add several missing methods to linuxfileops used by the OFED shim
layer: fo_write(), fo_truncate(), fo_kqfilter(), and fo_stat(). Most
of these used invfo_*(), but a dummy fo_stat() implementation was
added.
This will allow us to more easily test the software versions of these
routines...
Considering that we've never had an software asymetric implmentation,
it's doubtful anyone has this enabled...
swcr_newsession can change the pointer for swcr_sessions which races with
swcr_process which is looking up entries in this array.
Add a rwlock that protects changes to the array pointer so that
swcr_newsession and swcr_process no longer race.
Original patch by: Steve O'Hara-Smith <Steve.OHaraSmith@isilon.com>
Reviewed by: jmg
Sponsored by: EMC / Isilon Storage Division
In its stead use the Solaris / illumos approach of emulating '-' (dash)
in probe names with '__' (two consecutive underscores).
Reviewed by: markj
MFC after: 3 weeks
option, unbreak the lock tracing release semantic by embedding
calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined
version of the releasing functions for mutex, rwlock and sxlock.
Failing to do so skips the lockstat_probe_func invokation for
unlocking.
- As part of the LOCKSTAT support is inlined in mutex operation, for
kernel compiled without lock debugging options, potentially every
consumer must be compiled including opt_kdtrace.h.
Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the
dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES
is linked there and it is only used as a compile-time stub [0].
[0] immediately shows some new bug as DTRACE-derived support for debug
in sfxge is broken and it was never really tested. As it was not
including correctly opt_kdtrace.h before it was never enabled so it
was kept broken for a while. Fix this by using a protection stub,
leaving sfxge driver authors the responsibility for fixing it
appropriately [1].
Sponsored by: EMC / Isilon storage division
Discussed with: rstone
[0] Reported by: rstone
[1] Discussed with: philip
to implement fchown(2) and fchmod(2) support for several file types
that previously lacked it. Add MAC entries for chown/chmod done on
posix shared memory and (old) in-kernel posix semaphores.
Based on the submission by: glebius
Reviewed by: rwatson
Approved by: re (bz)
probe method return BUS_PROBE_NOWILDCARD so it doesn't get attached to real
devices hanging off of nexus(4) with no specific devclass set. Actually, the
more desirable fix for this would be to get rid of the newbus interface of
cryptosoft(4) altogether but apparently crypto(9) was written with support
for cryptographic hardware in mind so that approach would require some KPI
breaking changes which don't seem worth it.
MFC after: 1 week
use '-' in probe names, matching the probe names in Solaris.[1]
Add userland SDT probes definitions to sys/sdt.h.
Sponsored by: The FreeBSD Foundation
Discussed with: rwaston [1]
context from in-kernel execution of padlock instructions and to handle
spurious FPUDNA exceptions that sometime are raised when doing padlock
calculations.
Globally mark crypto(9) kthread as using FPU.
Reviewed by: pjd
Hardware provided by: Sentex Communications
Tested by: pho
PR: amd64/135014
MFC after: 1 month
how hashed MD5/SHA are implemented, abusing Final() for padding and
sw_octx to transport the key from the beginning to the end.
Enlightened about what was going on here by: cperciva
Reviewed by: cperciva
MFC After: 3 days
X-MFC with: r187826
PR: kern/126468
the compression was useless as well. Make sure to not update the data
and return, else we would waste resources when decompressing.
This also avoids the copyback() changing data other consumers like
xform_ipcomp.c would have ignored because of no win and sent out without
noting that compression was used, resulting in invalid packets at the
receiver.
MFC after: 5 days
This is needed to avoid running into out of buffer situations
where we cannot alloc a new buffer because we hit the array size
limit (ZBUF).
Use a combined allocation for the struct and the actual data buffer
to not increase the number of malloc calls. [1]
Defer initialization of zbuf until we actually need it.
Make sure the output buffer will be large enough in all cases.
Details discussed with: kib [1]
Reviewed by: kib [1]
MFC after: 6 days
replacement but only use it for inflate. For deflate use Z_FINISH
as Z_SYNC_FLUSH adds a trailing marker in some cases that inflate(),
despite the comment in zlib, does npt seem to cope well with, resulting
in errors when uncompressing exactly fills the outbut buffer without
a Z_STREAM_END and a successive call returns an error.
MFC after: 6 days
the return context, but from the original context.
Before repeating operation clear DONE flag and error.
Reviewed by: sam
Obtained from: Wheel Sp. z o.o. (http://www.wheel.pl)
Cryptodev uses UIO structure do get data from userspace and pass it to
cryptographic engines. Initially UIO size is equal to size of data passed to
engine, but if UIO is prepared for hash calculation an additional small space
is created to hold result of operation.
While creating space for the result, UIO I/O vector size is correctly
extended, but uio_resid field in UIO structure is not modified.
As bus_dma code uses uio_resid field to determine size of UIO DMA mapping,
resulting mapping hasn't correct size. This leads to a crash if all the
following conditions are met:
1. Hardware cryptographic accelerator writes result of hash operation
using DMA.
2. Size of input data is less or equal than (n * PAGE_SIZE),
3. Size of input data plus size of hash result is grather than
(n * PAGE_SIZE, where n is the same as in point 2.
This patch fixes this problem by adding size of the extenstion to uio_resid
field in UIO structure.
Submitted by: Piotr Ziecik kosmo ! semihalf dot com
Reviewed by: philip
Obtained from: Semihalf
can cope with a result buffer of NULL in the "Final" function, we cannot.
Thus pass in a temporary buffer long enough for either md5 or sha1 results
so that we do not panic.
PR: bin/126468
MFC after: 1 week
This makes it possible to support ftruncate() on non-vnode file types in
the future.
- 'struct fileops' grows a 'fo_truncate' method to handle an ftruncate() on
a given file descriptor.
- ftruncate() moves to kern/sys_generic.c and now just fetches a file
object and invokes fo_truncate().
- The vnode-specific portions of ftruncate() move to vn_truncate() in
vfs_vnops.c which implements fo_truncate() for vnode file types.
- Non-vnode file types return EINVAL in their fo_truncate() method.
Submitted by: rwatson
- Introduce a finit() which is used to initailize the fields of struct file
in such a way that the ops vector is only valid after the data, type,
and flags are valid.
- Protect f_flag and f_count with atomic operations.
- Remove the global list of all files and associated accounting.
- Rewrite the unp garbage collection such that it no longer requires
the global list of all files and instead uses a list of all unp sockets.
- Mark sockets in the accept queue so we don't incorrectly gc them.
Tested by: kris, pho
to kproc_xxx as they actually make whole processes.
Thos makes way for us to add REAL kthread_create() and friends
that actually make theads. it turns out that most of these
calls actually end up being moved back to the thread version
when it's added. but we need to make this cosmetic change first.
I'd LOVE to do this rename in 7.0 so that we can eventually MFC the
new kthread_xxx() calls.
and flags with an sxlock. This leads to a significant and measurable
performance improvement as a result of access to shared locking for
frequent lookup operations, reduced general overhead, and reduced overhead
in the event of contention. All of these are imported for threaded
applications where simultaneous access to a shared file descriptor array
occurs frequently. Kris has reported 2x-4x transaction rate improvements
on 8-core MySQL benchmarks; smaller improvements can be expected for many
workloads as a result of reduced overhead.
- Generally eliminate the distinction between "fast" and regular
acquisisition of the filedesc lock; the plan is that they will now all
be fast. Change all locking instances to either shared or exclusive
locks.
- Correct a bug (pointed out by kib) in fdfree() where previously msleep()
was called without the mutex held; sx_sleep() is now always called with
the sxlock held exclusively.
- Universally hold the struct file lock over changes to struct file,
rather than the filedesc lock or no lock. Always update the f_ops
field last. A further memory barrier is required here in the future
(discussed with jhb).
- Improve locking and reference management in linux_at(), which fails to
properly acquire vnode references before using vnode pointers. Annotate
improper use of vn_fullpath(), which will be replaced at a future date.
In fcntl(), we conservatively acquire an exclusive lock, even though in
some cases a shared lock may be sufficient, which should be revisited.
The dropping of the filedesc lock in fdgrowtable() is no longer required
as the sxlock can be held over the sleep operation; we should consider
removing that (pointed out by attilio).
Tested by: kris
Discussed with: jhb, kris, attilio, jeff
o make all crypto drivers have a device_t; pseudo drivers like the s/w
crypto driver synthesize one
o change the api between the crypto subsystem and drivers to use kobj;
cryptodev_if.m defines this api
o use the fact that all crypto drivers now have a device_t to add support
for specifying which of several potential devices to use when doing
crypto operations
o add new ioctls that allow user apps to select a specific crypto device
to use (previous ioctls maintained for compatibility)
o overhaul crypto subsystem code to eliminate lots of cruft and hide
implementation details from drivers
o bring in numerous fixes from Michale Richardson/hifn; mostly for
795x parts
o add an optional mechanism for mmap'ing the hifn 795x public key h/w
to user space for use by openssl (not enabled by default)
o update crypto test tools to use new ioctl's and add cmd line options
to specify a device to use for tests
These changes will also enable much future work on improving the core
crypto subsystem; including proper load balancing and interposing code
between the core and drivers to dispatch small operations to the s/w
driver as appropriate.
These changes were instigated by the work of Michael Richardson.
Reviewed by: pjd
Approved by: re
Before the change if a hardware crypto driver was loaded after
the software crypto driver, calling crypto_newsession() with
hard=0, will always choose software crypto.
- Add defines with block length for each HMAC algorithm.
- Add AES_BLOCK_LEN define which is an alias for RIJNDAEL128_BLOCK_LEN.
- Add NULL_BLOCK_LEN define.
Checking if the queues are empty is not enough for the crypto_proc thread
(it is enough for the crypto_ret_thread), because drivers can be marked
as blocked. In a situation where we have operations related to different
crypto drivers in the queue, it is possible that one driver is marked as
blocked. In this case, the queue will not be empty and we won't wakeup
the crypto_proc thread to execute operations for the others drivers.
Simply setting a global variable to 1 when we goes to sleep and setting
it back to 0 when we wake up is sufficient. The variable is protected
with the queue lock.
Before the change if the thread was working on symmetric operation, we
would send unnecessary wakeup after adding asymmetric operation (when
asym queue was empty) and vice versa.
twice if we call crypto_kinvoke() from crypto_proc thread.
This change also removes unprotected access to cc_kqblocked field
(CRYPTO_Q_LOCK() should be used for protection).
where crypto_invoke() returns ERESTART and before we set cc_qblocked to 1,
crypto_unblock() is called and sets it to 0. This way we mark device as
blocked forever.
Fix it by not setting cc_qblocked in the fast path and by protecting
crypto_invoke() in the crypto_proc thread with CRYPTO_Q_LOCK().
This won't slow things down, because there is no contention - we have
only one crypto thread. Actually it can be slightly faster, because we
save two atomic ops per crypto request.
The fast code path remains lock-less.
or SHA512, the blocksize is 128 bytes, not 64 bytes as anywhere else.
The bug also exists in NetBSD, OpenBSD and various other independed
implementations I look at.
- We cannot decide which hash function to use for HMAC based on the key
length, because any HMAC function can use any key length.
To fix it split CRYPTO_SHA2_HMAC into three algorithm:
CRYPTO_SHA2_256_HMAC, CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.
Those names are consistent with OpenBSD's naming.
- Remove authsize field from auth_hash structure.
- Allow consumer to define size of hash he wants to receive.
This allows to use HMAC not only for IPsec, where 96 bits MAC is requested.
The size of requested MAC is defined at newsession time in the cri_mlen
field - when 0, entire MAC will be returned.
- Add swcr_authprepare() function which prepares authentication key.
- Allow to provide key for every authentication operation, not only at
newsession time by honoring CRD_F_KEY_EXPLICIT flag.
- Make giving key at newsession time optional - don't try to operate on it
if its NULL.
- Extend COPYBACK()/COPYDATA() macros to handle CRYPTO_BUF_CONTIG buffer
type as well.
- Accept CRYPTO_BUF_IOV buffer type in swcr_authcompute() as we have
cuio_apply() now.
- 16 bits for key length (SW_klen) is more than enough.
Reviewed by: sam
crypto_invoke(). This allows to serve multiple crypto requests in
parallel and not bached requests are served lock-less.
Drivers should not depend on the queue lock beeing held around
crypto_invoke() and if they do, that's an error in the driver - it
should do its own synchronization.
- Don't forget to wakeup the crypto thread when new requests is
queued and only if both symmetric and asymmetric queues are empty.
- Symmetric requests use sessions and there is no way driver can
disappear when there is an active session, so we don't need to check
this, but assert this. This is also safe to not use the driver lock
in this case.
- Assymetric requests don't use sessions, so don't check the driver
in crypto_kinvoke().
- Protect assymetric operation with the driver lock, because if there
is no symmetric session, driver can disappear.
- Don't send assymetric request to the driver if it is marked as
blocked.
- Add an XXX comment, because I don't think migration to another driver
is safe when there are pending requests using freed session.
- Remove 'hint' argument from crypto_kinvoke(), as it serves no purpose.
- Don't hold the driver lock around kprocess method call, instead use
cc_koperations to track number of in-progress requests.
- Cleanup register/unregister code a bit.
- Other small simplifications and cleanups.
Reviewed by: sam
- Implement CUIO_SKIP() macro which is only responsible for skipping the given
number of bytes from iovec list. This allows to avoid duplicating the same
code in three functions.
Reviewed by: sam
and want to have crypto support loaded as KLD. By moving zlib to separate
module and adding MODULE_DEPEND directives, it is possible to use such
configuration without complication. Otherwise, since IPSEC is linked with
zlib (just like crypto.ko) you'll get following error:
interface zlib.1 already present in the KLD 'kernel'!
Approved by: cognet (mentor)
This is actually a local DoS, as every user can use /dev/crypto if there
is crypto hardware in the system and cryptodev.ko is loaded (or compiled
into the kernel).
Reported by: Mike Tancsa <mike@sentex.net>
MFC after: 1 day
It checked other algorithms against this bug and it seems they aren't
affected.
Reported by: Mike Tancsa <mike@sentex.net>
PR: i386/84860
Reviewed by: phk, cperciva(x2)
Don't grab Giant in the upper syscall/wrapper code
NET_LOCK_GIANT in the socket code (sockets/fifos).
mtx_lock(&Giant) in the vnode code.
mtx_lock(&Giant) in the opencrypto code. (This may actually not be
needed, but better safe than sorry).
Devfs grabs Giant if the driver is marked as needing Giant.
individual file object implementations can optionally acquire Giant if
they require it:
- soo_close(): depends on debug.mpsafenet
- pipe_close(): Giant not acquired
- kqueue_close(): Giant required
- vn_close(): Giant required
- cryptof_close(): Giant required (conservative)
Notes:
Giant is still acquired in close() even when closing MPSAFE objects
due to kqueue requiring Giant in the calling closef() code.
Microbenchmarks indicate that this removal of Giant cuts 3%-3% off
of pipe create/destroy pairs from user space with SMP compiled into
the kernel.
The cryptodev and opencrypto code appears MPSAFE, but I'm unable to
test it extensively and so have left Giant over fo_close(). It can
probably be removed given some testing and review.
Introduce d_version field in struct cdevsw, this must always be
initialized to D_VERSION.
Flip sense of D_NOGIANT flag to D_NEEDGIANT, this involves removing
four D_NOGIANT flags and adding 145 D_NEEDGIANT flags.
table, acquiring the necessary locks as it works. It usually returns
two references to the new descriptor: one in the descriptor table
and one via a pointer argument.
As falloc releases the FILEDESC lock before returning, there is a
potential for a process to close the reference in the file descriptor
table before falloc's caller gets to use the file. I don't think this
can happen in practice at the moment, because Giant indirectly protects
closes.
To stop the file being completly closed in this situation, this change
makes falloc set the refcount to two when both references are returned.
This makes life easier for several of falloc's callers, because the
first thing they previously did was grab an extra reference on the
file.
Reviewed by: iedowse
Idea run past: jhb
provide no methods does not make any sense, and is not used by any
driver.
It is a pretty hard to come up with even a theoretical concept of
a device driver which would always fail open and close with ENODEV.
Change the defaults to be nullopen() and nullclose() which simply
does nothing.
Remove explicit initializations to these from the drivers which
already used them.
software crypto device:
o record crypto device capabilities in each session id
o add a capability that indicates if the crypto driver operates synchronously
o tag the software crypto driver as operating synchronously
This commit also introduces crypto session id macros that cleanup their
construction and querying.
o add a ``done'' flag for crypto operations; this is set when the operation
completes and is intended for callers to check operations that may complete
``prematurely'' because of direct callbacks
o close a race for operations where the crypto driver returns ERESTART: we
need to hold the q lock to insure the blocked state for the driver and any
driver-private state is consistent; otherwise drivers may take an interrupt
and notify the crypto subsystem that it can unblock the driver but operations
will be left queued and never be processed
o close a race in /dev/crypto where operations can complete before the caller
can sleep waiting for the callback: use a per-session mutex and the new done
flag to handle this
o correct crypto_dispatch's handling of operations where the driver returns
ERESTART: the return value must be zero and not ERESTART, otherwise the
caller may free the crypto request despite it being queued for later handling
(this typically results in a later panic)
o change crypto mutex ``names'' so witness printouts and the like are more
meaningful
branches:
Initialize struct cdevsw using C99 sparse initializtion and remove
all initializations to default values.
This patch is automatically generated and has been tested by compiling
LINT with all the fields in struct cdevsw in reverse order on alpha,
sparc64 and i386.
Approved by: re(scottl)
should be done in crypto_done rather than in the callback thread
o use this flag to mark operations from /dev/crypto since the callback
routine just does a wakeup; this eliminates the last unneeded ctx switch
o change CRYPTO_F_NODELAY to CRYPTO_F_BATCH with an inverted meaning
so "0" becomes the default/desired setting (needed for user-mode
compatibility with openbsd)
o change crypto_dispatch to honor CRYPTO_F_BATCH instead of always
dispatching immediately
o remove uses of CRYPTO_F_NODELAY
o define COP_F_BATCH for ops submitted through /dev/crypto and pass
this on to the op that is submitted
Similar changes and more eventually coming for asymmetric ops.
MFC if re gives approval.
pointer types, and remove a huge number of casts from code using it.
Change struct xfile xf_data to xun_data (ABI is still compatible).
If we need to add a #define for f_data and xf_data we can, but I don't
think it will be necessary. There are no operational changes in this
commit.
or crypto_kdispatch unless the driver is currently blocked. This eliminates
the context switch to the dispatch thread for virtually all requests.
Note that this change means that for software crypto drivers the caller
will now block until the request is completed and the callback is dispatched
to the callback thread (h/w drivers will typically just dispatch the op to
the device and return quickly). If this is an issue we can either implement
a non-blocking interface in the s/w crypto driver or use either the
"no delay" flag in the crypto request or the "software driver" capability
flag to control what to do.
Sponsored by: Vernier Networks
cryptodev or kldunload cryptodev module); crypto statistcs; remove
unused alloctype field from crypto op to offset addition of the
performance time stamp
Supported by: Vernier Networks
for processing callbacks. This closes race conditions caused by locking
too many things with a single mutex.
o reclaim crypto requests under certain (impossible) failure conditions
a consistent interface to h/w and s/w crypto algorithms for use by the
kernel and (for h/w at least) by user-mode apps. Access for user-level
code is through a /dev/crypto device that'll eventually be used by openssl
to (potentially) accelerate many applications. Coming soon is an IPsec
that makes use of this service to accelerate ESP, AH, and IPCOMP protocols.
Included here is the "core" crypto support, /dev/crypto driver, various
crypto algorithms that are not already present in the KAME crypto area,
and support routines used by crypto device drivers.
Obtained from: openbsd
