Commit Graph

30 Commits

Author SHA1 Message Date
Konstantin Belousov
04c49e68de Use the fpu_kern_enter() interface to properly separate usermode FPU
context from in-kernel execution of padlock instructions and to handle
spurious FPUDNA exceptions that sometime are raised when doing padlock
calculations.

Globally mark crypto(9) kthread as using FPU.

Reviewed by:	pjd
Hardware provided by:	Sentex Communications
Tested by:	  pho
PR:    amd64/135014
MFC after:    1 month
2010-06-05 16:00:53 +00:00
Bjoern A. Zeeb
df21ad6e41 Define an SDT provider for "opencrypto".
MFC after:	6 days
2009-11-28 16:54:18 +00:00
Julian Elischer
3745c395ec Rename the kthread_xxx (e.g. kthread_create()) calls
to kproc_xxx as they actually make whole processes.
Thos makes way for us to add REAL kthread_create() and friends
that actually make theads. it turns out that most of these
calls actually end up being moved back to the thread version
when it's added. but we need to make this cosmetic change first.

I'd LOVE to do this rename in 7.0  so that we can eventually MFC the
new kthread_xxx() calls.
2007-10-20 23:23:23 +00:00
Sam Leffler
6810ad6f2a Overhaul driver/subsystem api's:
o make all crypto drivers have a device_t; pseudo drivers like the s/w
  crypto driver synthesize one
o change the api between the crypto subsystem and drivers to use kobj;
  cryptodev_if.m defines this api
o use the fact that all crypto drivers now have a device_t to add support
  for specifying which of several potential devices to use when doing
  crypto operations
o add new ioctls that allow user apps to select a specific crypto device
  to use (previous ioctls maintained for compatibility)
o overhaul crypto subsystem code to eliminate lots of cruft and hide
  implementation details from drivers
o bring in numerous fixes from Michale Richardson/hifn; mostly for
  795x parts
o add an optional mechanism for mmap'ing the hifn 795x public key h/w
  to user space for use by openssl (not enabled by default)
o update crypto test tools to use new ioctl's and add cmd line options
  to specify a device to use for tests

These changes will also enable much future work on improving the core
crypto subsystem; including proper load balancing and interposing code
between the core and drivers to dispatch small operations to the s/w
driver as appropriate.

These changes were instigated by the work of Michael Richardson.

Reviewed by:	pjd
Approved by:	re
2007-03-21 03:42:51 +00:00
Pawel Jakub Dawidek
0d5c337bef When DIAGNOSTIC is defined, verify if we don't free crypto requests from
the crypto queue or from the return queue.
2006-06-06 15:04:52 +00:00
Pawel Jakub Dawidek
694e011306 Prefer hardware crypto over software crypto.
Before the change if a hardware crypto driver was loaded after
the software crypto driver, calling crypto_newsession() with
hard=0, will always choose software crypto.
2006-06-04 22:12:08 +00:00
Pawel Jakub Dawidek
3a865c827a Improve the code responsible for waking up the crypto_proc thread.
Checking if the queues are empty is not enough for the crypto_proc thread
(it is enough for the crypto_ret_thread), because drivers can be marked
as blocked. In a situation where we have operations related to different
crypto drivers in the queue, it is possible that one driver is marked as
blocked. In this case, the queue will not be empty and we won't wakeup
the crypto_proc thread to execute operations for the others drivers.

Simply setting a global variable to 1 when we goes to sleep and setting
it back to 0 when we wake up is sufficient. The variable is protected
with the queue lock.
2006-05-22 10:05:23 +00:00
Pawel Jakub Dawidek
9c12ca29d6 Don't wakeup the crypto_ret_proc thread if it is running already.
Before the change if the thread was working on symmetric operation, we
would send unnecessary wakeup after adding asymmetric operation (when
asym queue was empty) and vice versa.
2006-05-22 09:58:34 +00:00
Pawel Jakub Dawidek
04d8f36a4f Don't set cc_kqblocked twice and don't increment cryptostats.cs_kblocks
twice if we call crypto_kinvoke() from crypto_proc thread.
This change also removes unprotected access to cc_kqblocked field
(CRYPTO_Q_LOCK() should be used for protection).
2006-05-22 09:37:28 +00:00
Pawel Jakub Dawidek
bda0abc627 We must synchronize access to cc_qblocked, because there could be a race
where crypto_invoke() returns ERESTART and before we set cc_qblocked to 1,
crypto_unblock() is called and sets it to 0. This way we mark device as
blocked forever.

Fix it by not setting cc_qblocked in the fast path and by protecting
crypto_invoke() in the crypto_proc thread with CRYPTO_Q_LOCK().
This won't slow things down, because there is no contention - we have
only one crypto thread. Actually it can be slightly faster, because we
save two atomic ops per crypto request.
The fast code path remains lock-less.
2006-05-22 07:48:45 +00:00
Pawel Jakub Dawidek
c3c820369e Silent Coverity Prevent report by asserting that cap != NULL.
Coverity ID:	1414
2006-05-18 06:28:39 +00:00
Pawel Jakub Dawidek
4acae0ac29 - Make opencrypto more SMP friendly by dropping the queue lock around
crypto_invoke(). This allows to serve multiple crypto requests in
  parallel and not bached requests are served lock-less.
  Drivers should not depend on the queue lock beeing held around
  crypto_invoke() and if they do, that's an error in the driver - it
  should do its own synchronization.
- Don't forget to wakeup the crypto thread when new requests is
  queued and only if both symmetric and asymmetric queues are empty.
- Symmetric requests use sessions and there is no way driver can
  disappear when there is an active session, so we don't need to check
  this, but assert this. This is also safe to not use the driver lock
  in this case.
- Assymetric requests don't use sessions, so don't check the driver
  in crypto_kinvoke().
- Protect assymetric operation with the driver lock, because if there
  is no symmetric session, driver can disappear.
- Don't send assymetric request to the driver if it is marked as
  blocked.
- Add an XXX comment, because I don't think migration to another driver
  is safe when there are pending requests using freed session.
- Remove 'hint' argument from crypto_kinvoke(), as it serves no purpose.
- Don't hold the driver lock around kprocess method call, instead use
  cc_koperations to track number of in-progress requests.
- Cleanup register/unregister code a bit.
- Other small simplifications and cleanups.

Reviewed by:	sam
2006-05-17 18:12:44 +00:00
Pawel Jakub Dawidek
71af8134f7 Be sure to wakeup the crypto thread when new request was queued.
This should fix a hang when starting cryptokeytest (and more).

MFC after:	1 month
2006-04-11 18:01:04 +00:00
Wojciech A. Koszek
0a0eb0e8db crypto.ko depends on zlib.
Submitted by:	Ben Kelly <bkelly at vadev.org>
Approved by:	rwatson
Point hat to:	me
MFC after:	1 day
2006-03-04 15:50:46 +00:00
Warner Losh
60727d8b86 /* -> /*- for license, minor formatting changes 2005-01-07 02:29:27 +00:00
Poul-Henning Kamp
5dba30f15a add missing #include <sys/module.h> 2004-05-30 20:27:19 +00:00
John Baldwin
6074439965 kthread_exit() no longer requires Giant, so don't force callers to acquire
Giant just to call kthread_exit().

Requested by:	many
2004-03-05 22:42:17 +00:00
Sam Leffler
d8409aaf6e consolidate callback optimization check in one location by adding a flag
for crypto operations that indicates the crypto code should do the check
in crypto_done

MFC after:	1 day
2003-06-30 05:09:32 +00:00
Sam Leffler
07d0c94a46 Add support to eliminate a context switch per crypto op when using the
software crypto device:

o record crypto device capabilities in each session id
o add a capability that indicates if the crypto driver operates synchronously
o tag the software crypto driver as operating synchronously

This commit also introduces crypto session id macros that cleanup their
construction and querying.
2003-06-27 20:07:10 +00:00
David E. O'Brien
2c44651495 Use __FBSDID(). 2003-06-11 05:57:50 +00:00
Sam Leffler
3569ae7f66 Flush my local cache of cryto subsystem fixes:
o add a ``done'' flag for crypto operations; this is set when the operation
  completes and is intended for callers to check operations that may complete
  ``prematurely'' because of direct callbacks
o close a race for operations where the crypto driver returns ERESTART: we
  need to hold the q lock to insure the blocked state for the driver and any
  driver-private state is consistent; otherwise drivers may take an interrupt
  and notify the crypto subsystem that it can unblock the driver but operations
  will be left queued and never be processed
o close a race in /dev/crypto where operations can complete before the caller
  can sleep waiting for the callback: use a per-session mutex and the new done
  flag to handle this
o correct crypto_dispatch's handling of operations where the driver returns
  ERESTART: the return value must be zero and not ERESTART, otherwise the
  caller may free the crypto request despite it being queued for later handling
  (this typically results in a later panic)
o change crypto mutex ``names'' so witness printouts and the like are more
  meaningful
2003-06-02 23:28:06 +00:00
Sam Leffler
eb73a605cd o add a CRYPTO_F_CBIMM flag to symmetric ops to indicate the callback
should be done in crypto_done rather than in the callback thread
o use this flag to mark operations from /dev/crypto since the callback
  routine just does a wakeup; this eliminates the last unneeded ctx switch
o change CRYPTO_F_NODELAY to CRYPTO_F_BATCH with an inverted meaning
  so "0" becomes the default/desired setting (needed for user-mode
  compatibility with openbsd)
o change crypto_dispatch to honor CRYPTO_F_BATCH instead of always
  dispatching immediately
o remove uses of CRYPTO_F_NODELAY
o define COP_F_BATCH for ops submitted through /dev/crypto and pass
  this on to the op that is submitted

Similar changes and more eventually coming for asymmetric ops.

MFC if re gives approval.
2003-02-23 07:25:48 +00:00
Sam Leffler
f7890744d0 Invoke the driver directly when a request is submitted via crypto_dispatch
or crypto_kdispatch unless the driver is currently blocked.  This eliminates
the context switch to the dispatch thread for virtually all requests.

Note that this change means that for software crypto drivers the caller
will now block until the request is completed and the callback is dispatched
to the callback thread (h/w drivers will typically just dispatch the op to
the device and return quickly).  If this is an issue we can either implement
a non-blocking interface in the s/w crypto driver or use either the
"no delay" flag in the crypto request or the "software driver" capability
flag to control what to do.

Sponsored by:	Vernier Networks
2003-01-09 05:39:04 +00:00
Sam Leffler
bc0c6d3c99 correct flags passed when allocation crypto request structures so we
don't block and the returned data is zero'd
2003-01-06 18:52:05 +00:00
Sam Leffler
51e45326ca manage kernel threads properly; especially shutting them down on module unload
Reviewed by:	jhb
2003-01-03 06:23:46 +00:00
Sam Leffler
7d1853ee0e MFS: crypto timing support; purge usercrypto sysctl (just don't config
cryptodev or kldunload cryptodev module); crypto statistcs; remove
unused alloctype field from crypto op to offset addition of the
performance time stamp

Supported by:	Vernier Networks
2003-01-03 06:16:59 +00:00
Sam Leffler
5a7294d14e change load order so module is present before crypto drivers
Submitted by:	Doug Ambrisko" <ambrisko@verniernetworks.com>
2002-11-08 23:07:03 +00:00
Mark Murray
f544a52873 Module-ize the 'core' crypto stuff. This may still need to be compiled
into the kernel by default (if required), but other modules can now
depend() on this.

Fix inter-module dependancy.

Earlier version OK'ed by:	sam
2002-10-16 14:31:34 +00:00
Sam Leffler
1a91ccccc6 o split crypto_proc into two threads: one for processing requests and one
for processing callbacks.  This closes race conditions caused by locking
  too many things with a single mutex.
o reclaim crypto requests under certain (impossible) failure conditions
2002-10-07 18:46:38 +00:00
Sam Leffler
091d81d134 In-kernel crypto framework derived from openbsd. This facility provides
a consistent interface to h/w and s/w crypto algorithms for use by the
kernel and (for h/w at least) by user-mode apps.  Access for user-level
code is through a /dev/crypto device that'll eventually be used by openssl
to (potentially) accelerate many applications.  Coming soon is an IPsec
that makes use of this service to accelerate ESP, AH, and IPCOMP protocols.

Included here is the "core" crypto support, /dev/crypto driver, various
crypto algorithms that are not already present in the KAME crypto area,
and support routines used by crypto device drivers.

Obtained from:	openbsd
2002-10-04 20:31:23 +00:00