John Baldwin 470e851c4b ktls: Support asynchronous dispatch of AEAD ciphers. ... KTLS OCF support was originally targeted at software backends that used host CPU cycles to encrypt TLS records. As a result, each KTLS worker thread queued a single TLS record at a time and waited for it to be encrypted before processing another TLS record. This works well for software backends but limits throughput on OCF drivers for coprocessors that support asynchronous operation such as qat(4) or ccr(4). This change uses an alternate function (ktls_encrypt_async) when encrypt TLS records via a coprocessor. This function queues TLS records for encryption and returns. It defers the work done after a TLS record has been encrypted (such as marking the mbufs ready) to a callback invoked asynchronously by the coprocessor driver when a record has been encrypted. - Add a struct ktls_ocf_state that holds the per-request state stored on the stack for synchronous requests. Asynchronous requests malloc this structure while synchronous requests continue to allocate this structure on the stack. - Add a ktls_encrypt_async() variant of ktls_encrypt() which does not perform request completion after dispatching a request to OCF. Instead, the ktls_ocf backends invoke ktls_encrypt_cb() when a TLS record request completes for an asynchronous request. - Flag AEAD software TLS sessions as async if the backend driver selected by OCF is an async driver. - Pull code to create and dispatch an OCF request out of ktls_encrypt() into a new ktls_encrypt_one() function used by both ktls_encrypt() and ktls_encrypt_async(). - Pull code to "finish" the VM page shuffling for a file-backed TLS record into a helper function ktls_finish_noanon() used by both ktls_encrypt() and ktls_encrypt_cb(). Reviewed by: markj Tested on: ccr(4) (jhb), qat(4) (markj) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D31665		2021-08-30 13:11:52 -07:00
..
bus_if.m	newbus: Move from bus_child_{pnpinfo,location}_src to bus_child_{pnpinfo,location} with sbuf	2021-06-22 20:52:06 -06:00
capabilities.conf	Add fspacectl(2), vn_deallocate(9) and VOP_DEALLOCATE(9).	2021-08-05 23:20:42 +08:00
clock_if.m
cpufreq_if.m
device_if.m
firmw.S	Use a template assembly file for firmware object files.	2020-12-17 20:31:17 +00:00
genassym.sh	genassym.sh: Fix two minor issues found by shellcheck	2021-07-28 13:49:16 -06:00
genoffset.c
genoffset.sh	genoffset: simplify and rewrite in sh	2021-07-28 13:50:09 -06:00
imgact_aout.c	fork: Allow ABI to specify fork return values for child.	2021-08-12 11:45:25 +03:00
imgact_binmisc.c	imgact_binmisc: limit the extent of match on incoming entries	2020-11-08 04:24:29 +00:00
imgact_elf32.c
imgact_elf64.c
imgact_elf.c	Create sys/reg.h for the common code previously in machine/reg.h	2021-08-30 12:50:53 +01:00
imgact_shell.c
init_main.c	fork: Allow ABI to specify fork return values for child.	2021-08-12 11:45:25 +03:00
init_sysent.c	Regen after 0dc332bff2	2021-08-05 23:22:02 +08:00
kern_acct.c	acct: Zero pad bytes in accounting records	2021-07-23 10:29:57 -04:00
kern_alq.c
kern_clock.c
kern_clocksource.c
kern_condvar.c	Use sleepq_signal(SLEEPQ_DROP) in cv_signal().	2021-07-09 20:57:58 -04:00
kern_conf.c	Fix handling of D_GIANTOK	2021-06-07 16:45:50 -04:00
kern_cons.c
kern_context.c
kern_cpu.c
kern_cpuset.c	domainset: Define additional global policies	2021-04-14 13:03:33 -04:00
kern_ctf.c
kern_descrip.c	fsetown: Avoid process group lock recursion	2021-08-28 15:50:44 -04:00
kern_dtrace.c	dtrace: stop using eventhandlers for the part compiled into the kernel	2020-11-23 18:27:21 +00:00
kern_dump.c	Always use 64-bit physical addresses for dump_avail[] in minidumps	2020-12-03 17:12:31 +00:00
kern_environment.c	kenv: allow listing of static kernel environments	2021-07-18 23:06:19 -05:00
kern_et.c	Remove NO_EVENTTIMERS support	2020-11-19 02:50:48 +00:00
kern_event.c	kqueue: replace kq_ncallouts loop with atomic_fetchadd	2021-06-02 15:14:58 +00:00
kern_exec.c	Create sys/reg.h for the common code previously in machine/reg.h	2021-08-30 12:50:53 +01:00
kern_exit.c	umtx: Split umtx.h on two counterparts.	2021-07-29 12:41:29 +03:00
kern_fail.c
kern_ffclock.c
kern_fork.c	fork: Remove the unnecessary spaces.	2021-08-12 11:58:17 +03:00
kern_hhook.c
kern_idle.c
kern_intr.c	Mark some sysctls as CTLFLAG_MPSAFE.	2021-08-10 22:18:26 -04:00
kern_jail.c	kern: ether_gen_addr: randomize on default hostuuid, too	2021-06-01 22:59:21 -05:00
kern_kcov.c	Generalize bus_space(9) and atomic(9) sanitizer interceptors	2021-03-22 22:21:53 -04:00
kern_khelp.c
kern_kthread.c	umtx: Split umtx.h on two counterparts.	2021-07-29 12:41:29 +03:00
kern_ktr.c
kern_ktrace.c	ktrace: Zero request structures when populating the pool	2021-07-23 10:29:53 -04:00
kern_linker.c	link_elf_obj: Invoke fini callbacks	2021-07-29 09:46:25 -04:00
kern_lock.c	lockprof: pass lock type as an argument instead of reading the spin flag	2021-05-23 17:55:27 +00:00
kern_lockf.c
kern_lockstat.c
kern_loginclass.c	Drop "All rights reserved" from all my stuff. This includes	2020-10-28 13:46:11 +00:00
kern_malloc.c	uma: Add KMSAN hooks	2021-08-10 21:27:54 -04:00
kern_mbuf.c	Introduce m_get3()	2021-08-18 08:48:27 +02:00
kern_mib.c	jail: Consistently handle the pr_allow bitmask	2020-12-26 20:25:02 -08:00
kern_module.c
kern_mtxpool.c
kern_mutex.c	Fix lockstat:::thread-spin dtrace probe with LOCK_PROFILING	2021-08-02 14:44:23 -05:00
kern_ntptime.c	Add kern_ntp_adjtime(9).	2020-12-04 18:56:44 +00:00
kern_osd.c
kern_physio.c	Minor style tidy: if( -> if (	2021-04-18 11:19:15 -06:00
kern_pmc.c
kern_poll.c
kern_priv.c	jail: Consistently handle the pr_allow bitmask	2020-12-26 20:25:02 -08:00
kern_proc.c	kdb: Handle process enumeration before procinit()	2021-08-11 14:44:22 -03:00
kern_procctl.c	procctl(PROC_ASLR_STATUS): fix vmspace leak	2021-07-15 03:02:50 +03:00
kern_prot.c	kern: fail getgroup and setgroup with negative int	2021-06-02 13:22:57 -06:00
kern_racct.c	umtx: Split umtx.h on two counterparts.	2021-07-29 12:41:29 +03:00
kern_rangelock.c
kern_rctl.c	rctl(4): support throttling resource usage to 0	2021-01-11 15:36:57 -08:00
kern_resource.c	umtx: Split umtx.h on two counterparts.	2021-07-29 12:41:29 +03:00
kern_rmlock.c	rmlock(9): add an RM_DUPOK flag	2021-04-12 11:42:21 -03:00
kern_rwlock.c	lockprof: pass lock type as an argument instead of reading the spin flag	2021-05-23 17:55:27 +00:00
kern_sdt.c
kern_sema.c
kern_sendfile.c	Fix handling of errors from pru_send(PRUS_NOTREADY)	2021-05-21 17:45:19 -04:00
kern_sharedpage.c	Remove bogus cast from exec_sysvec_init().	2021-07-20 09:54:09 +03:00
kern_shutdown.c	netdump: send key before dump, in case dump fails	2021-08-11 10:54:56 -05:00
kern_sig.c	sigtimedwait: Use a unique wait channel for sleeping	2021-08-16 15:11:15 -04:00
kern_switch.c
kern_sx.c	lockprof: pass lock type as an argument instead of reading the spin flag	2021-05-23 17:55:27 +00:00
kern_synch.c	tsleep: Add a PNOLOCK flag	2021-08-05 17:16:30 -04:00
kern_syscalls.c	Move syscall_thread_{enter,exit}() into the slow path. This is only	2020-11-08 15:54:59 +00:00
kern_sysctl.c	Escape any '.' characters in sysctl node names	2021-07-22 10:22:48 -06:00
kern_tc.c	Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights	2021-08-08 10:42:24 -04:00
kern_thr.c	umtx: Split umtx.h on two counterparts.	2021-07-29 12:41:29 +03:00
kern_thread.c	kern: Ensure that thread-local KMSAN state is available	2021-08-10 21:27:53 -04:00
kern_time.c	posix timers: Check for overflow when converting to ns	2021-05-13 08:34:03 -04:00
kern_timeout.c	callout: Make cc_cpu local to kern_timeout.c	2021-07-15 22:41:10 -04:00
kern_tslog.c	kern_tslog: Include tslog data from loader	2021-06-20 20:09:47 -07:00
kern_ubsan.c
kern_umtx.c	umtx: Split do_unlock_pi on two counterparts.	2021-07-29 12:47:39 +03:00
kern_uuid.c
kern_xxx.c
ksched.c
link_elf_obj.c	link_elf_obj: Invoke fini callbacks	2021-07-29 09:46:25 -04:00
link_elf.c	vm_map_protect: allow to set prot and max_prot in one go.	2021-01-13 01:35:22 +02:00
linker_if.m
Make.tags.inc
Makefile
md4c.c
md5c.c
msi_if.m	Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights	2021-08-08 10:42:24 -04:00
p1003_1b.c
pic_if.m
posix4_mib.c
sched_4bsd.c	umtx: Split umtx.h on two counterparts.	2021-07-29 12:41:29 +03:00
sched_ule.c	sched_ule(4): Pre-seed sched_random().	2021-08-02 10:55:28 -04:00
serdev_if.m
stack_protector.c
subr_acl_nfs4.c	Drop "All rights reserved" from all my stuff. This includes	2020-10-28 13:46:11 +00:00
subr_acl_posix1e.c
subr_asan.c	Add interceptors for atomic operations on userspace memory	2021-07-29 21:14:36 -04:00
subr_atomic64.c
subr_autoconf.c	config_intrhook: provide config_intrhook_drain	2021-03-11 09:45:10 -07:00
subr_blist.c	blist: Correct the node count computed in blist_create()	2021-07-13 17:47:27 -04:00
subr_boot.c
subr_bufring.c
subr_bus_dma.c	busdma: Add KMSAN integration	2021-08-10 21:27:54 -04:00
subr_bus.c	devclass_alloc_unit: move "at" hint test to after device-in-use test	2021-08-02 11:27:17 -05:00
subr_busdma_bufalloc.c
subr_capability.c	Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights	2021-08-08 10:42:24 -04:00
subr_clock.c
subr_compressor.c
subr_counter.c	Rationalize per-cpu zones.	2020-11-05 15:08:56 +00:00
subr_coverage.c	Generalize bus_space(9) and atomic(9) sanitizer interceptors	2021-03-22 22:21:53 -04:00
subr_csan.c	Simplify kernel sanitizer interceptors	2021-07-29 21:13:32 -04:00
subr_devmap.c	ddb: fix show devmap output on 32-bit arm	2021-02-18 11:53:14 -04:00
subr_devstat.c	Speed up geom_stats_resync in the presence of many devices	2021-03-02 18:33:45 -07:00
subr_disk.c	Enable bioq 'car limit' added at r335066 at 128 bios.	2020-10-26 04:04:06 +00:00
subr_dummy_vdso_tc.c
subr_early.c
subr_epoch.c	The old thread priority must be stored as part of the EPOCH(9) tracker.	2021-05-23 10:53:25 +02:00
subr_eventhandler.c
subr_fattime.c
subr_filter.c
subr_firmware.c	firmware(9): extend firmware_get() by a "no warn" flag.	2021-01-27 13:51:26 +00:00
subr_gtaskqueue.c	Import kernel WireGuard support	2020-11-29 19:38:03 +00:00
subr_hash.c
subr_hints.c	Optimize res_find().	2021-08-08 21:54:49 -04:00
subr_intr.c	intrng: remove now redundant shadow variable.	2021-07-08 08:46:41 +02:00
subr_kdb.c	kdb: Handle process enumeration before procinit()	2021-08-11 14:44:22 -03:00
subr_kobj.c	newbus: Optimize/Simplify kobj_class_compile_common a little	2021-04-21 15:37:24 -06:00
subr_lock.c	lockprof: move panic check after inspecting the state	2021-05-23 17:55:27 +00:00
subr_log.c	Document kern.log_wakeups_per_second.	2021-08-04 11:50:34 -07:00
subr_mchain.c
subr_module.c	sysctl debug.dump_modinfo should recognize font module	2021-01-08 09:24:49 +02:00
subr_msan.c	kmsan: Export kmsan_mark_mbuf() and kmsan_mark_bio()	2021-08-11 16:33:41 -04:00
subr_msgbuf.c
subr_param.c	Move mips and arm to 1000Hz by default.	2021-06-16 20:00:14 -06:00
subr_pcpu.c	Add more per-cpu zones.	2020-11-09 00:34:23 +00:00
subr_pctrie.c
subr_physmem.c	Add support for hw.physmem tunable for ARM/ARM64/RISC-V platforms	2020-12-03 05:39:27 +00:00
subr_pidctrl.c
subr_power.c
subr_prf.c	kvprintf(9): add missing FALLTHROUGH	2021-01-22 00:18:40 +01:00
subr_prng.c
subr_prof.c	x86: remove gcov kernel support	2021-04-02 15:41:51 +03:00
subr_rangeset.c	Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights	2021-08-08 10:42:24 -04:00
subr_rman.c	rman: Remove an outdated comment that no longer applies	2021-07-05 16:15:03 +01:00
subr_rtc.c
subr_sbuf.c	sbuf_uionew(): sbuf_new() takes int as length	2021-04-14 10:23:20 +03:00
subr_scanf.c
subr_sfbuf.c
subr_sglist.c	sglist: Add sglist_append_single_mbuf().	2021-05-25 16:59:18 -07:00
subr_sleepqueue.c	Allow sleepq_signal() to drop the lock.	2021-06-25 14:12:21 -04:00
subr_smp.c	Refactor/optimize cpu_search_*().	2021-07-28 22:00:29 -04:00
subr_smr.c
subr_stack.c
subr_stats.c	stats(3): Improve t-digest merging of samples which result in mu adjustment underflow.	2021-04-02 13:17:53 +11:00
subr_syscall.c	Pass the syscall number to capsicum permission-denied signals	2021-07-16 18:06:44 +01:00
subr_taskqueue.c
subr_terminal.c	Remove "All Rights Reserved" from FreeBSD Foundation sys/ copyrights	2021-08-08 10:42:24 -04:00
subr_trap.c	amd64: Add MD bits for KMSAN	2021-08-10 21:27:53 -04:00
subr_turnstile.c	Use atomic loads/stores when updating td->td_state	2021-02-18 14:02:48 +00:00
subr_uio.c
subr_unit.c
subr_vmem.c	vmem: Revert r364744	2020-12-01 16:06:31 +00:00
subr_witness.c	witness: remove ifnet_rw	2021-08-18 08:51:26 +02:00
sys_capability.c
sys_eventfd.c	Drop "All rights reserved" from my copyright statements.	2021-07-20 10:05:50 +03:00
sys_generic.c	Add fspacectl(2), vn_deallocate(9) and VOP_DEALLOCATE(9).	2021-08-05 23:20:42 +08:00
sys_getrandom.c
sys_pipe.c	pipe_paircreate(): do not leak pipepair memory on error	2021-08-16 17:08:44 +03:00
sys_procdesc.c	procdesc: convert the zone to a malloc type	2020-11-09 00:05:21 +00:00
sys_process.c	Create sys/reg.h for the common code previously in machine/reg.h	2021-08-30 12:50:53 +01:00
sys_socket.c	Re-enable network ioctls in capability mode	2021-04-23 09:22:49 -04:00
syscalls.c	Regen after 0dc332bff2	2021-08-05 23:22:02 +08:00
syscalls.master	Add fspacectl(2), vn_deallocate(9) and VOP_DEALLOCATE(9).	2021-08-05 23:20:42 +08:00
systrace_args.c	Regen after 0dc332bff2	2021-08-05 23:22:02 +08:00
sysv_ipc.c
sysv_msg.c	jail: Change the locking around pr_ref and pr_uref	2021-02-21 10:55:44 -08:00
sysv_sem.c	jail: Change the locking around pr_ref and pr_uref	2021-02-21 10:55:44 -08:00
sysv_shm.c	jail: Change the locking around pr_ref and pr_uref	2021-02-21 10:55:44 -08:00
tty_compat.c
tty_info.c
tty_inq.c
tty_outq.c	Minor style cleanup	2021-04-18 11:14:17 -06:00
tty_pts.c
tty_tty.c
tty_ttydisc.c
tty.c	ttydev_write: prevent stops while terminal is busied	2021-05-18 20:52:03 +03:00
uipc_accf.c	Consistently use the SOLISTENING() macro	2021-06-14 17:32:27 -04:00
uipc_debug.c
uipc_domain.c	uipc: avoid circular pr_{slow,fast}timos	2021-08-18 12:46:54 -05:00
uipc_ktls.c	ktls: Support asynchronous dispatch of AEAD ciphers.	2021-08-30 13:11:52 -07:00
uipc_mbuf2.c
uipc_mbuf.c	m_dup: Handle unmapped mbufs as an input mbuf.	2021-07-26 14:09:16 -07:00
uipc_mbufhash.c	Implement mbuf hashing routines for IP over infiniband, IPoIB.	2020-10-22 09:17:56 +00:00
uipc_mqueue.c	jail: Change the locking around pr_ref and pr_uref	2021-02-21 10:55:44 -08:00
uipc_sem.c	Convert remaining cap_rights_init users to cap_rights_init_one	2021-01-12 13:16:10 +00:00
uipc_shm.c	fspacectl(2): Changes on rmsr.r_offset's minimum value returned	2021-08-26 00:03:37 +08:00
uipc_sockbuf.c	socket: Implement SO_RERROR	2021-07-28 09:35:09 -07:00
uipc_socket.c	socket: Implement SO_RERROR	2021-07-28 09:35:09 -07:00
uipc_syscalls.c	connect: Use soconnectat() unconditionally in kern_connect()	2021-08-27 08:32:07 -04:00
uipc_usrreq.c	socket: Implement SO_RERROR	2021-07-28 09:35:09 -07:00
vfs_acl.c
vfs_aio.c	lio_listio(2): Allow LIO_READV and LIO_WRITEV.	2021-08-22 23:00:42 +12:00
vfs_bio.c	amd64: Populate the KMSAN shadow maps and integrate with the VM	2021-08-10 21:27:53 -04:00
vfs_cache.c	cache: retire cache_fast_revlookup sysctl	2021-08-23 15:31:44 +02:00
vfs_cluster.c	Minor style tidy: if( -> if (	2021-04-18 11:19:15 -06:00
vfs_default.c	vfs_default: Change vop_stddeallocate() from static to global	2021-08-27 18:25:44 -07:00
vfs_export.c
vfs_extattr.c
vfs_hash.c	vfs_hash_rehash(): require the vnode to be exclusively locked	2021-08-27 18:39:45 +03:00
vfs_init.c	VFS_QUOTACTL(9): allow implementation to indicate busy state changes	2021-05-30 14:53:47 -07:00
vfs_lookup.c	vfs: s/__unused/__diagused in crossmp_*	2021-08-23 15:23:42 +02:00
vfs_mount.c	vfs: drop dedicated sysinit for mountlist_mtx	2021-08-26 20:52:03 +02:00
vfs_mountroot.c	vfs: Initialize "lastfail" in vfs_mountroot_wait()	2021-07-23 12:04:02 -04:00
vfs_subr.c	VFS: remove MNTK_MARKER	2021-07-24 12:52:32 -07:00
vfs_syscalls.c	fstatat(2): handle non-vnode file descriptors for AT_EMPTY_PATH	2021-08-14 00:17:18 +03:00
vfs_vnops.c	vfs: yield in vn_deallocate_impl() loop	2021-08-29 16:26:00 +08:00
vnode_if.src	vfs: Add ioflag to VOP_DEALLOCATE(9)	2021-08-12 23:03:49 +08:00