c034143269
- The linked list of cryptoini structures used in session initialization is replaced with a new flat structure: struct crypto_session_params. This session includes a new mode to define how the other fields should be interpreted. Available modes include: - COMPRESS (for compression/decompression) - CIPHER (for simply encryption/decryption) - DIGEST (computing and verifying digests) - AEAD (combined auth and encryption such as AES-GCM and AES-CCM) - ETA (combined auth and encryption using encrypt-then-authenticate) Additional modes could be added in the future (e.g. if we wanted to support TLS MtE for AES-CBC in the kernel we could add a new mode for that. TLS modes might also affect how AAD is interpreted, etc.) The flat structure also includes the key lengths and algorithms as before. However, code doesn't have to walk the linked list and switch on the algorithm to determine which key is the auth key vs encryption key. The 'csp_auth_*' fields are always used for auth keys and settings and 'csp_cipher_*' for cipher. (Compression algorithms are stored in csp_cipher_alg.) - Drivers no longer register a list of supported algorithms. This doesn't quite work when you factor in modes (e.g. a driver might support both AES-CBC and SHA2-256-HMAC separately but not combined for ETA). Instead, a new 'crypto_probesession' method has been added to the kobj interface for symmteric crypto drivers. This method returns a negative value on success (similar to how device_probe works) and the crypto framework uses this value to pick the "best" driver. There are three constants for hardware (e.g. ccr), accelerated software (e.g. aesni), and plain software (cryptosoft) that give preference in that order. One effect of this is that if you request only hardware when creating a new session, you will no longer get a session using accelerated software. Another effect is that the default setting to disallow software crypto via /dev/crypto now disables accelerated software. Once a driver is chosen, 'crypto_newsession' is invoked as before. - Crypto operations are now solely described by the flat 'cryptop' structure. The linked list of descriptors has been removed. A separate enum has been added to describe the type of data buffer in use instead of using CRYPTO_F_* flags to make it easier to add more types in the future if needed (e.g. wired userspace buffers for zero-copy). It will also make it easier to re-introduce separate input and output buffers (in-kernel TLS would benefit from this). Try to make the flags related to IV handling less insane: - CRYPTO_F_IV_SEPARATE means that the IV is stored in the 'crp_iv' member of the operation structure. If this flag is not set, the IV is stored in the data buffer at the 'crp_iv_start' offset. - CRYPTO_F_IV_GENERATE means that a random IV should be generated and stored into the data buffer. This cannot be used with CRYPTO_F_IV_SEPARATE. If a consumer wants to deal with explicit vs implicit IVs, etc. it can always generate the IV however it needs and store partial IVs in the buffer and the full IV/nonce in crp_iv and set CRYPTO_F_IV_SEPARATE. The layout of the buffer is now described via fields in cryptop. crp_aad_start and crp_aad_length define the boundaries of any AAD. Previously with GCM and CCM you defined an auth crd with this range, but for ETA your auth crd had to span both the AAD and plaintext (and they had to be adjacent). crp_payload_start and crp_payload_length define the boundaries of the plaintext/ciphertext. Modes that only do a single operation (COMPRESS, CIPHER, DIGEST) should only use this region and leave the AAD region empty. If a digest is present (or should be generated), it's starting location is marked by crp_digest_start. Instead of using the CRD_F_ENCRYPT flag to determine the direction of the operation, cryptop now includes an 'op' field defining the operation to perform. For digests I've added a new VERIFY digest mode which assumes a digest is present in the input and fails the request with EBADMSG if it doesn't match the internally-computed digest. GCM and CCM already assumed this, and the new AEAD mode requires this for decryption. The new ETA mode now also requires this for decryption, so IPsec and GELI no longer do their own authentication verification. Simple DIGEST operations can also do this, though there are no in-tree consumers. To eventually support some refcounting to close races, the session cookie is now passed to crypto_getop() and clients should no longer set crp_sesssion directly. - Assymteric crypto operation structures should be allocated via crypto_getkreq() and freed via crypto_freekreq(). This permits the crypto layer to track open asym requests and close races with a driver trying to unregister while asym requests are in flight. - crypto_copyback, crypto_copydata, crypto_apply, and crypto_contiguous_subsegment now accept the 'crp' object as the first parameter instead of individual members. This makes it easier to deal with different buffer types in the future as well as separate input and output buffers. It's also simpler for driver writers to use. - bus_dmamap_load_crp() loads a DMA mapping for a crypto buffer. This understands the various types of buffers so that drivers that use DMA do not have to be aware of different buffer types. - Helper routines now exist to build an auth context for HMAC IPAD and OPAD. This reduces some duplicated work among drivers. - Key buffers are now treated as const throughout the framework and in device drivers. However, session key buffers provided when a session is created are expected to remain alive for the duration of the session. - GCM and CCM sessions now only specify a cipher algorithm and a cipher key. The redundant auth information is not needed or used. - For cryptosoft, split up the code a bit such that the 'process' callback now invokes a function pointer in the session. This function pointer is set based on the mode (in effect) though it simplifies a few edge cases that would otherwise be in the switch in 'process'. It does split up GCM vs CCM which I think is more readable even if there is some duplication. - I changed /dev/crypto to support GMAC requests using CRYPTO_AES_NIST_GMAC as an auth algorithm and updated cryptocheck to work with it. - Combined cipher and auth sessions via /dev/crypto now always use ETA mode. The COP_F_CIPHER_FIRST flag is now a no-op that is ignored. This was actually documented as being true in crypto(4) before, but the code had not implemented this before I added the CIPHER_FIRST flag. - I have not yet updated /dev/crypto to be aware of explicit modes for sessions. I will probably do that at some point in the future as well as teach it about IV/nonce and tag lengths for AEAD so we can support all of the NIST KAT tests for GCM and CCM. - I've split up the exising crypto.9 manpage into several pages of which many are written from scratch. - I have converted all drivers and consumers in the tree and verified that they compile, but I have not tested all of them. I have tested the following drivers: - cryptosoft - aesni (AES only) - blake2 - ccr and the following consumers: - cryptodev - IPsec - ktls_ocf - GELI (lightly) I have not tested the following: - ccp - aesni with sha - hifn - kgssapi_krb5 - ubsec - padlock - safe - armv8_crypto (aarch64) - glxsb (i386) - sec (ppc) - cesa (armv7) - cryptocteon (mips64) - nlmsec (mips64) Discussed with: cem Relnotes: yes Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D23677
207 lines
8.1 KiB
C
207 lines
8.1 KiB
C
/*-
|
|
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
|
|
*
|
|
* Copyright (c) 2003 Sam Leffler, Errno Consulting
|
|
* Copyright (c) 2003 Global Technology Associates, Inc.
|
|
* All rights reserved.
|
|
*
|
|
* Redistribution and use in source and binary forms, with or without
|
|
* modification, are permitted provided that the following conditions
|
|
* are met:
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
* notice, this list of conditions and the following disclaimer.
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
* documentation and/or other materials provided with the distribution.
|
|
*
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
* SUCH DAMAGE.
|
|
*
|
|
* $FreeBSD$
|
|
*/
|
|
#ifndef _SAFE_SAFEVAR_H_
|
|
#define _SAFE_SAFEVAR_H_
|
|
|
|
/* Maximum queue length */
|
|
#ifndef SAFE_MAX_NQUEUE
|
|
#define SAFE_MAX_NQUEUE 60
|
|
#endif
|
|
|
|
#define SAFE_MAX_PART 64 /* Maximum scatter/gather depth */
|
|
#define SAFE_DMA_BOUNDARY 0 /* No boundary for source DMA ops */
|
|
#define SAFE_MAX_DSIZE MCLBYTES /* Fixed scatter particle size */
|
|
#define SAFE_MAX_SSIZE 0x0ffff /* Maximum gather particle size */
|
|
#define SAFE_MAX_DMA 0xfffff /* Maximum PE operand size (20 bits) */
|
|
/* total src+dst particle descriptors */
|
|
#define SAFE_TOTAL_DPART (SAFE_MAX_NQUEUE * SAFE_MAX_PART)
|
|
#define SAFE_TOTAL_SPART (SAFE_MAX_NQUEUE * SAFE_MAX_PART)
|
|
|
|
#define SAFE_RNG_MAXBUFSIZ 128 /* 32-bit words */
|
|
|
|
#define SAFE_DEF_RTY 0xff /* PCI Retry Timeout */
|
|
#define SAFE_DEF_TOUT 0xff /* PCI TRDY Timeout */
|
|
#define SAFE_DEF_CACHELINE 0x01 /* Cache Line setting */
|
|
|
|
#ifdef _KERNEL
|
|
/*
|
|
* State associated with the allocation of each chunk
|
|
* of memory setup for DMA.
|
|
*/
|
|
struct safe_dma_alloc {
|
|
u_int32_t dma_paddr; /* physical address */
|
|
caddr_t dma_vaddr; /* virtual address */
|
|
bus_dma_tag_t dma_tag; /* bus dma tag used */
|
|
bus_dmamap_t dma_map; /* associated map */
|
|
bus_dma_segment_t dma_seg;
|
|
bus_size_t dma_size; /* mapped memory size (bytes) */
|
|
int dma_nseg; /* number of segments */
|
|
};
|
|
|
|
/*
|
|
* Cryptographic operand state. One of these exists for each
|
|
* source and destination operand passed in from the crypto
|
|
* subsystem. When possible source and destination operands
|
|
* refer to the same memory. More often they are distinct.
|
|
* We track the virtual address of each operand as well as
|
|
* where each is mapped for DMA.
|
|
*/
|
|
struct safe_operand {
|
|
bus_dmamap_t map;
|
|
bus_size_t mapsize;
|
|
int nsegs;
|
|
bus_dma_segment_t segs[SAFE_MAX_PART];
|
|
};
|
|
|
|
/*
|
|
* Packet engine ring entry and cryptographic operation state.
|
|
* The packet engine requires a ring of descriptors that contain
|
|
* pointers to various cryptographic state. However the ring
|
|
* configuration register allows you to specify an arbitrary size
|
|
* for ring entries. We use this feature to collect most of the
|
|
* state for each cryptographic request into one spot. Other than
|
|
* ring entries only the ``particle descriptors'' (scatter/gather
|
|
* lists) and the actual operand data are kept separate. The
|
|
* particle descriptors must also be organized in rings. The
|
|
* operand data can be located aribtrarily (modulo alignment constraints).
|
|
*
|
|
* Note that the descriptor ring is mapped onto the PCI bus so
|
|
* the hardware can DMA data. This means the entire ring must be
|
|
* contiguous.
|
|
*/
|
|
struct safe_ringentry {
|
|
struct safe_desc re_desc; /* command descriptor */
|
|
struct safe_sarec re_sa; /* SA record */
|
|
struct safe_sastate re_sastate; /* SA state record */
|
|
struct cryptop *re_crp; /* crypto operation */
|
|
|
|
struct safe_operand re_src; /* source operand */
|
|
struct safe_operand re_dst; /* destination operand */
|
|
struct mbuf *re_dst_m;
|
|
|
|
int unused;
|
|
int re_flags;
|
|
#define SAFE_QFLAGS_COPYOUTICV 0x2 /* copy back on completion */
|
|
};
|
|
|
|
#define re_src_map re_src.map
|
|
#define re_src_nsegs re_src.nsegs
|
|
#define re_src_segs re_src.segs
|
|
#define re_src_mapsize re_src.mapsize
|
|
|
|
#define re_dst_map re_dst.map
|
|
#define re_dst_nsegs re_dst.nsegs
|
|
#define re_dst_segs re_dst.segs
|
|
#define re_dst_mapsize re_dst.mapsize
|
|
|
|
struct rndstate_test;
|
|
|
|
struct safe_session {
|
|
u_int32_t ses_klen; /* key length in bits */
|
|
u_int32_t ses_key[8]; /* DES/3DES/AES key */
|
|
u_int32_t ses_mlen; /* hmac length in bytes */
|
|
u_int32_t ses_hminner[5]; /* hmac inner state */
|
|
u_int32_t ses_hmouter[5]; /* hmac outer state */
|
|
};
|
|
|
|
struct safe_softc {
|
|
device_t sc_dev; /* device backpointer */
|
|
struct resource *sc_irq;
|
|
void *sc_ih; /* interrupt handler cookie */
|
|
bus_space_handle_t sc_sh; /* memory handle */
|
|
bus_space_tag_t sc_st; /* memory tag */
|
|
struct resource *sc_sr; /* memory resource */
|
|
bus_dma_tag_t sc_srcdmat; /* source dma tag */
|
|
bus_dma_tag_t sc_dstdmat; /* destination dma tag */
|
|
u_int sc_chiprev; /* major/minor chip revision */
|
|
int sc_flags; /* device specific flags */
|
|
#define SAFE_FLAGS_KEY 0x01 /* has key accelerator */
|
|
#define SAFE_FLAGS_RNG 0x02 /* hardware rng */
|
|
int sc_suspended;
|
|
int sc_needwakeup; /* notify crypto layer */
|
|
int32_t sc_cid; /* crypto tag */
|
|
uint32_t sc_devinfo;
|
|
struct safe_dma_alloc sc_ringalloc; /* PE ring allocation state */
|
|
struct safe_ringentry *sc_ring; /* PE ring */
|
|
struct safe_ringentry *sc_ringtop; /* PE ring top */
|
|
struct safe_ringentry *sc_front; /* next free entry */
|
|
struct safe_ringentry *sc_back; /* next pending entry */
|
|
int sc_nqchip; /* # passed to chip */
|
|
struct mtx sc_ringmtx; /* PE ring lock */
|
|
struct safe_pdesc *sc_spring; /* src particle ring */
|
|
struct safe_pdesc *sc_springtop; /* src particle ring top */
|
|
struct safe_pdesc *sc_spfree; /* next free src particle */
|
|
struct safe_dma_alloc sc_spalloc; /* src particle ring state */
|
|
struct safe_pdesc *sc_dpring; /* dest particle ring */
|
|
struct safe_pdesc *sc_dpringtop; /* dest particle ring top */
|
|
struct safe_pdesc *sc_dpfree; /* next free dest particle */
|
|
struct safe_dma_alloc sc_dpalloc; /* dst particle ring state */
|
|
|
|
struct callout sc_rngto; /* rng timeout */
|
|
struct rndtest_state *sc_rndtest; /* RNG test state */
|
|
void (*sc_harvest)(struct rndtest_state *,
|
|
void *, u_int);
|
|
};
|
|
#endif /* _KERNEL */
|
|
|
|
struct safe_stats {
|
|
u_int64_t st_ibytes;
|
|
u_int64_t st_obytes;
|
|
u_int32_t st_ipackets;
|
|
u_int32_t st_opackets;
|
|
u_int32_t st_invalid; /* invalid argument */
|
|
u_int32_t st_badsession; /* invalid session id */
|
|
u_int32_t st_badflags; /* flags indicate !(mbuf | uio) */
|
|
u_int32_t st_nodesc; /* op submitted w/o descriptors */
|
|
u_int32_t st_badalg; /* unsupported algorithm */
|
|
u_int32_t st_ringfull; /* PE descriptor ring full */
|
|
u_int32_t st_peoperr; /* PE marked error */
|
|
u_int32_t st_dmaerr; /* PE DMA error */
|
|
u_int32_t st_bypasstoobig; /* bypass > 96 bytes */
|
|
u_int32_t st_skipmismatch; /* enc part begins before auth part */
|
|
u_int32_t st_lenmismatch; /* enc length different auth length */
|
|
u_int32_t st_coffmisaligned; /* crypto offset not 32-bit aligned */
|
|
u_int32_t st_cofftoobig; /* crypto offset > 255 words */
|
|
u_int32_t st_iovmisaligned; /* iov op not aligned */
|
|
u_int32_t st_iovnotuniform; /* iov op not suitable */
|
|
u_int32_t st_unaligned; /* unaligned src caused copy */
|
|
u_int32_t st_notuniform; /* non-uniform src caused copy */
|
|
u_int32_t st_nomap; /* bus_dmamap_create failed */
|
|
u_int32_t st_noload; /* bus_dmamap_load_* failed */
|
|
u_int32_t st_nombuf; /* MGET* failed */
|
|
u_int32_t st_nomcl; /* MCLGET* failed */
|
|
u_int32_t st_maxqchip; /* max mcr1 ops out for processing */
|
|
u_int32_t st_rng; /* RNG requests */
|
|
u_int32_t st_rngalarm; /* RNG alarm requests */
|
|
u_int32_t st_noicvcopy; /* ICV data copies suppressed */
|
|
};
|
|
#endif /* _SAFE_SAFEVAR_H_ */
|