Alexander Motin 3db35ffa2a Some more taskqueue optimizations. ... - Optimize enqueue for two task priority values by adding new tq_hint field, pointing to the last task inserted into the middle of the list. In case of more then two priority values it should halve average search. - Move tq_active insert/remove out of the taskqueue_run_locked loop. Instead of dirtying few shared cache lines per task introduce different mechanism to drain active tasks, based on task sequence number counter, that uses only cache lines already present in cache. Since the new mechanism does not need ordering, switch tq_active from TAILQ to LIST. - Move static and dynamic struct taskqueue fields into different cache lines. Move lock into its own cache line, so that heavy lock spinning by multiple waiting threads would not affect the running thread. - While there, correct some TQ_SLEEP() wait messages. This change fixes certain ZFS write workloads, causing huge congestion on taskqueue lock. Those workloads combine some large block writes to saturate the pool and trigger allocation throttling, which uses higher priority tasks to requeue the delayed I/Os, with many small blocks to generate deep queue of small tasks for taskqueue to sort. MFC after: 1 week Sponsored by: iXsystems, Inc.		2019-11-01 22:49:44 +00:00
..
bus_if.m	Add necessary bits for Linux KPI to work correctly on powerpc	2019-08-04 19:28:10 +00:00
capabilities.conf	Add a shm_open2 syscall to support upcoming memfd_create	2019-09-25 17:59:15 +00:00
clock_if.m
cpufreq_if.m
device_if.m
genassym.sh
genoffset.c	Merge td_epochnest with td_no_sleeping.	2019-10-29 17:28:25 +00:00
genoffset.sh
imgact_aout.c	Switch to use shared vnode locks for text files during image activation.	2019-05-05 11:20:43 +00:00
imgact_binmisc.c
imgact_elf32.c
imgact_elf64.c
imgact_elf.c	avoid kernel stack data leak in core dump thrmisc note	2019-10-31 20:42:36 +00:00
imgact_shell.c
init_main.c	Remove epoch tracker from struct thread. It was an ugly crutch to emulate	2019-10-21 18:19:32 +00:00
init_sysent.c	sysent: regenerate after r352747.	2019-09-26 15:41:10 +00:00
kern_acct.c
kern_alq.c
kern_clock.c	Update td_runtime of running thread on each statclock().	2019-06-14 01:09:10 +00:00
kern_clocksource.c
kern_condvar.c
kern_conf.c	Remove the remnants of SI_CHEAPCLONE	2019-10-05 21:52:06 +00:00
kern_cons.c
kern_context.c
kern_cpu.c
kern_cpuset.c	Remove a redundant NULL pointer check in cpuset_modify_domain().	2019-09-12 16:47:38 +00:00
kern_ctf.c	Convert DDB_CTF to use newer version of ZLIB.	2019-08-08 07:27:49 +00:00
kern_descrip.c	Disallow fcntl(F_READAHEAD) when the vnode is not a regular file.	2019-10-02 15:45:49 +00:00
kern_dtrace.c
kern_dump.c	[PPC64] Initial kernel minidump implementation	2019-10-14 13:04:04 +00:00
kern_environment.c	Follow up r352244: kenv: tighten up assertions	2019-09-12 14:34:46 +00:00
kern_et.c
kern_event.c	Avoid relying on header pollution from sys/refcount.h.	2019-07-29 20:26:01 +00:00
kern_exec.c	(4/6) Protect page valid with the busy lock.	2019-10-15 03:45:41 +00:00
kern_exit.c	proc: clear pid bitmap entry after dropping proctree lock	2019-09-02 12:46:43 +00:00
kern_fail.c
kern_ffclock.c
kern_fork.c	rfork(2): add RFSPAWN flag	2019-09-25 19:20:41 +00:00
kern_hhook.c
kern_idle.c
kern_intr.c	Disable intr_storm_threshold mechanism by default	2019-05-24 22:33:14 +00:00
kern_jail.c
kern_kcov.c	Change synchonization rules for vm_page reference counting.	2019-09-09 21:32:42 +00:00
kern_khelp.c
kern_kthread.c
kern_ktr.c
kern_ktrace.c
kern_linker.c
kern_lock.c	Add lockmgr(9) probes to the lockstat DTrace provider.	2019-08-21 23:43:58 +00:00
kern_lockf.c
kern_lockstat.c	Add lockmgr(9) probes to the lockstat DTrace provider.	2019-08-21 23:43:58 +00:00
kern_loginclass.c
kern_malloc.c	Merge td_epochnest with td_no_sleeping.	2019-10-29 17:28:25 +00:00
kern_mbuf.c	Split out a more generic debugnet(4) from netdump(4)	2019-10-17 16:23:03 +00:00
kern_mib.c	Add missing include of sys/boot.h	2019-06-24 20:52:21 +00:00
kern_module.c
kern_mtxpool.c
kern_mutex.c	INVARIANTS: treat LA_LOCKED as the same of LA_XLOCKED in mtx_assert.	2019-08-23 06:39:40 +00:00
kern_ntptime.c
kern_osd.c
kern_physio.c
kern_pmc.c
kern_poll.c
kern_priv.c
kern_proc.c	Remove page locking from pmap_mincore().	2019-10-16 22:03:27 +00:00
kern_procctl.c	Add procctl(PROC_STACKGAP_CTL)	2019-09-03 18:56:25 +00:00
kern_prot.c	Only enable COMPAT_43 changes for syscalls ABI for a.out processes.	2019-08-11 19:16:07 +00:00
kern_racct.c	proc: eliminate the zombproc list	2019-08-28 16:18:23 +00:00
kern_rangelock.c	rangelock: add rangelock_cookie_assert	2019-09-15 02:59:53 +00:00
kern_rctl.c
kern_resource.c
kern_rmlock.c
kern_rwlock.c
kern_sdt.c
kern_sema.c
kern_sendfile.c	Add a TOE KTLS mode and a TOE hook for allocating TLS sessions.	2019-10-08 21:34:06 +00:00
kern_sharedpage.c
kern_shutdown.c	debug,kassert.warnings is a statistic, not a tunable	2019-10-21 12:21:56 +00:00
kern_sig.c	rfork(2): add RFSPAWN flag	2019-09-25 19:20:41 +00:00
kern_switch.c
kern_sx.c
kern_synch.c	Merge td_epochnest with td_no_sleeping.	2019-10-29 17:28:25 +00:00
kern_syscalls.c
kern_sysctl.c	sysctl: use names instead of magic numbers.	2019-09-18 16:13:10 +00:00
kern_tc.c	Initialize timehands linkage much earlier.	2019-09-09 12:42:48 +00:00
kern_thr.c
kern_thread.c	amd64: move pcb out of kstack to struct thread.	2019-10-25 20:09:42 +00:00
kern_time.c	Disallow excessively small times of day in clock_settime(2).	2019-05-03 21:26:44 +00:00
kern_timeout.c	Allocate callout wheel from the respective memory domain.	2019-09-21 15:38:08 +00:00
kern_tslog.c
kern_ubsan.c	Teach the kernel KUBSAN runtime about alignment_assumption	2019-05-28 09:12:15 +00:00
kern_umtx.c	Make umtxq_check_susp() to correctly handle thread exit requests.	2019-08-01 14:34:27 +00:00
kern_uuid.c	Use sbuf_cat() in GEOM confxml generation.	2019-06-19 15:36:02 +00:00
kern_xxx.c
ksched.c
link_elf_obj.c	Apply mapping protections to preloaded kernel modules on amd64.	2019-10-18 13:56:45 +00:00
link_elf.c	Apply mapping protections to preloaded kernel modules on amd64.	2019-10-18 13:56:45 +00:00
linker_if.m
Make.tags.inc	Remove a couple of harmless stray references to nandfs.	2019-06-25 16:39:25 +00:00
Makefile
makesyscalls.sh	Restore the ability to set capenabled directly in syscalls.conf.	2019-09-30 20:58:29 +00:00
md4c.c
md5c.c
msi_if.m
p1003_1b.c
pic_if.m
posix4_mib.c
sched_4bsd.c	Reduce umtx-related work on exec and exit	2019-05-08 16:30:38 +00:00
sched_ule.c	Microoptimize sched_pickcpu() CPU affinity on SMT.	2019-09-26 00:35:06 +00:00
serdev_if.m
stack_protector.c	Revert r346292 (permit_nonrandom_stackcookies)	2019-05-13 23:37:44 +00:00
subr_acl_nfs4.c
subr_acl_posix1e.c
subr_autoconf.c
subr_blist.c	Address problems in blist_alloc introduced in r349777. The swap block allocator could become corrupted	2019-07-11 20:52:39 +00:00
subr_boot.c	When parsing command line stuff, treat tabs and spaces the same.	2019-04-18 22:52:12 +00:00
subr_bufring.c
subr_bus_dma.c	Add an external mbuf buffer type that holds multiple unmapped pages.	2019-06-29 00:48:33 +00:00
subr_bus.c	Allocate device softc from the device domain.	2019-10-12 19:03:07 +00:00
subr_busdma_bufalloc.c
subr_capability.c
subr_clock.c
subr_compressor.c	GZIO: Update to use zlib 1.2.11.	2019-08-25 07:50:44 +00:00
subr_counter.c
subr_coverage.c
subr_devmap.c	Generalize ARM specific comments in devmap	2019-10-15 23:21:52 +00:00
subr_devstat.c
subr_disk.c
subr_dummy_vdso_tc.c
subr_early.c
subr_epoch.c	Merge td_epochnest with td_no_sleeping.	2019-10-29 17:28:25 +00:00
subr_eventhandler.c	Include ktr.h in more compilation units	2019-05-21 20:38:48 +00:00
subr_fattime.c
subr_filter.c	This commit adds BBR (Bottleneck Bandwidth and RTT) congestion control. This	2019-09-24 18:18:11 +00:00
subr_firmware.c
subr_gtaskqueue.c	Some more taskqueue optimizations.	2019-11-01 22:49:44 +00:00
subr_hash.c
subr_hints.c
subr_intr.c	Extract eventfilter declarations to sys/_eventfilter.h	2019-05-20 00:38:23 +00:00
subr_kdb.c	Implement NetGDB(4)	2019-10-17 21:33:01 +00:00
subr_kobj.c
subr_lock.c	lockprof: move per-cpu data to dpcpu	2019-09-22 20:44:24 +00:00
subr_log.c
subr_mchain.c
subr_module.c
subr_msgbuf.c
subr_param.c	riscv: restore default HZ=1000, keep QEMU at HZ=100	2019-09-07 05:13:31 +00:00
subr_pcpu.c	vfs: manage mnt_ref with atomics	2019-09-16 21:31:02 +00:00
subr_pctrie.c
subr_pidctrl.c
subr_power.c	Extract eventfilter declarations to sys/_eventfilter.h	2019-05-20 00:38:23 +00:00
subr_prf.c	device_printf: Use sbuf for more coherent prints on SMP	2019-05-07 17:47:20 +00:00
subr_prof.c
subr_rangeset.c
subr_rman.c
subr_rtc.c
subr_sbuf.c	sbuf(9): Add sbuf_nl_terminate() API	2019-08-07 19:27:14 +00:00
subr_scanf.c	Add support for 'j', 't' and 'z' flags to kernel sscanf().	2019-08-16 19:46:22 +00:00
subr_sfbuf.c
subr_sglist.c	Add an external mbuf buffer type that holds multiple unmapped pages.	2019-06-29 00:48:33 +00:00
subr_sleepqueue.c	Merge td_epochnest with td_no_sleeping.	2019-10-29 17:28:25 +00:00
subr_smp.c	x86: Implement MWAIT support for stopping a CPU	2019-05-04 20:34:26 +00:00
subr_stack.c	Add debugging facility EPOCH_TRACE that checks that epochs entered are	2019-09-25 18:26:31 +00:00
subr_stats.c	Introduce stats(3), a flexible statistics gathering API.	2019-10-07 19:05:05 +00:00
subr_syscall.c	Don't pass error from syscallenter() to syscallret().	2019-07-15 21:25:16 +00:00
subr_taskqueue.c	Some more taskqueue optimizations.	2019-11-01 22:49:44 +00:00
subr_terminal.c	kernel terminal should initialize fg and bg variables before calling TUNABLE_INT_FETCH	2019-09-26 07:19:26 +00:00
subr_trap.c	Merge td_epochnest with td_no_sleeping.	2019-10-29 17:28:25 +00:00
subr_turnstile.c	Fix the turnstile_lock() KPI.	2019-07-24 23:04:59 +00:00
subr_uio.c
subr_unit.c
subr_vmem.c	Don't free the cursor boundary tag during vmem_destroy().	2019-10-09 21:20:39 +00:00
subr_witness.c	Only check the blessings table for known LORs.	2019-08-02 18:01:47 +00:00
sys_capability.c
sys_generic.c	devfs: plug redundant bwillwrite avoidance	2019-10-05 17:44:33 +00:00
sys_getrandom.c
sys_pipe.c	Modify pipe_poll() to properly check for pending direct writes.	2019-08-21 19:35:04 +00:00
sys_procdesc.c	procdesc: fix the function name	2019-08-05 20:31:17 +00:00
sys_process.c	Define macro VM_MAP_ENTRY_FOREACH for enumerating the entries in a vm_map.	2019-10-08 07:14:21 +00:00
sys_socket.c
syscalls.c	sysent: regenerate after r352747.	2019-09-26 15:41:10 +00:00
syscalls.master	sysalls.master: remove superfluous ellipsis in comment	2019-10-01 17:05:21 +00:00
systrace_args.c	sysent: regenerate after r352747.	2019-09-26 15:41:10 +00:00
sysv_ipc.c	sysv: get rid of fork/exit hooks if the code is compiled in	2019-05-04 19:05:30 +00:00
sysv_msg.c
sysv_sem.c
sysv_shm.c	sysv: get rid of fork/exit hooks if the code is compiled in	2019-05-04 19:05:30 +00:00
tty_compat.c
tty_info.c
tty_inq.c
tty_outq.c
tty_pts.c
tty_tty.c	Extract eventfilter declarations to sys/_eventfilter.h	2019-05-20 00:38:23 +00:00
tty_ttydisc.c
tty.c	Defer funsetown() calls for a TTY to tty_rel_free().	2019-07-04 15:42:02 +00:00
uipc_accf.c
uipc_debug.c
uipc_domain.c	Remove pfctlinput2(). It came from KAME and had never ever been in use.	2019-10-15 15:40:03 +00:00
uipc_ktls.c	Use a counter with a random base for explicit IVs in GCM.	2019-10-24 18:13:26 +00:00
uipc_mbuf2.c	In m_pulldown(), before trying to prepend bytes to the subsequent mbuf,	2019-08-09 05:18:59 +00:00
uipc_mbuf.c	kTLS: Fix a bug where we would not encrypt anon data inplace.	2019-09-27 20:08:19 +00:00
uipc_mbufhash.c
uipc_mqueue.c	mqueuefs: fix compat32 struct file leak	2019-08-20 17:44:03 +00:00
uipc_sem.c
uipc_shm.c	(4/6) Protect page valid with the busy lock.	2019-10-15 03:45:41 +00:00
uipc_sockbuf.c	Add kernel-side support for in-kernel TLS.	2019-08-27 00:01:56 +00:00
uipc_socket.c	Cleanup unneeded includes that crept in with r353292.	2019-10-09 16:59:42 +00:00
uipc_syscalls.c	Only enable COMPAT_43 changes for syscalls ABI for a.out processes.	2019-08-11 19:16:07 +00:00
uipc_usrreq.c	Fix handling of empty SCM_RIGHTS messages.	2019-10-08 23:34:48 +00:00
vfs_acl.c
vfs_aio.c
vfs_bio.c	Drop the object lock in vfs_bio and cluster where it is now safe to do so.	2019-10-29 20:37:59 +00:00
vfs_cache.c	cache: decrease ncnegfactor to 5	2019-09-27 19:14:03 +00:00
vfs_cluster.c	Drop the object lock in vfs_bio and cluster where it is now safe to do so.	2019-10-29 20:37:59 +00:00
vfs_default.c	vfs: apply r352437 to the fast path as well	2019-09-17 15:53:40 +00:00
vfs_export.c
vfs_extattr.c
vfs_hash.c	vfs: implement usecount implying holdcnt	2019-09-03 15:42:11 +00:00
vfs_init.c	vfs: add optional root vnode caching	2019-10-06 22:14:32 +00:00
vfs_lookup.c	NDFREE(): Fix unlocking for LOCKPARENT|LOCKLEAF and ndp->ni_dvp == ndp->ni_vp.	2019-05-21 15:12:13 +00:00
vfs_mount.c	vfs: add optional root vnode caching	2019-10-06 22:14:32 +00:00
vfs_mountroot.c	mountroot: run statfs after mounting devfs	2019-10-11 17:04:38 +00:00
vfs_subr.c	Replace OBJ_MIGHTBEDIRTY with a system using atomics. Remove the TMPFS_DIRTY	2019-10-29 21:06:34 +00:00
vfs_syscalls.c	vfs: stop always overwriting ->mnt_stat in VFS_STATFS	2019-08-18 18:40:12 +00:00
vfs_vnops.c	Stop leaking information from the kernel through timespec	2019-10-16 13:21:01 +00:00
vnode_if.src	vfs: add VOP_NEED_INACTIVE	2019-08-28 20:34:24 +00:00