mjg 2eae2c18c9 sx: port over writer starvation prevention measures from rwlock ...

A constant stream of readers could completely starve writers and this is not
a hypothetical scenario.

The 'poll2_threads' test from the will-it-scale suite reliably starves writers
even with concurrency < 10 threads.

The problem was run into and diagnosed by dillon@backplane.com

There was next to no change in lock contention profile during -j 128 pkg build,
despite an sx lock being at the top.

Tested by:	pho

2018-05-22 07:20:22 +00:00

..

bus_if.m

…

capabilities.conf

…

clock_if.m

…

cpufreq_if.m

…

device_if.m

…

genassym.sh

…

imgact_aout.c

i386 4/4G split.

2018-04-13 20:30:49 +00:00

imgact_binmisc.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

imgact_elf32.c

…

imgact_elf64.c

…

imgact_elf.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

imgact_gzip.c

Use C99 designated initializers for struct execsw

2018-03-13 13:09:10 +00:00

imgact_shell.c

Use C99 designated initializers for struct execsw

2018-03-13 13:09:10 +00:00

inflate.c

…

init_main.c

Use strsep() to parse init_path in start_init().

2018-05-17 23:07:51 +00:00

init_sysent.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

kern_acct.c

…

kern_alq.c

…

kern_clock.c

epoch: skip poll function call in hardclock unless there are callbacks pending

2018-05-17 21:39:15 +00:00

kern_clocksource.c

stop and restart kernel event timers in the suspend / resume cycle

2018-05-21 20:23:04 +00:00

kern_condvar.c

…

kern_conf.c

conf: revert last change and annotate unused var instead

2018-05-19 05:07:03 +00:00

kern_cons.c

…

kern_context.c

…

kern_cpu.c

Boost thread priority while changing CPU frequency

2018-05-07 15:24:03 +00:00

kern_cpuset.c

cpuset: revert and annotate instead

2018-05-19 05:07:31 +00:00

kern_ctf.c

…

kern_descrip.c

capsicum: propagate const correctness

2018-05-19 05:14:05 +00:00

kern_dtrace.c

systrace: track it like sdt probes

2018-04-27 15:16:34 +00:00

kern_dump.c

Print the dump progress indicator after calling dump_start().

2018-05-01 17:32:43 +00:00

kern_environment.c

Use explicit_bzero() when cleaning values out of the kernel environment.

2018-04-10 22:57:56 +00:00

kern_et.c

…

kern_event.c

kevent: annotate unused stack local

2018-05-19 05:06:18 +00:00

kern_exec.c

Eliminate the overhead of gratuitous repeated reinitialization of cap_rights

2018-05-09 18:47:24 +00:00

kern_exit.c

Add PROC_PDEATHSIG_SET to procctl interface.

2018-04-18 21:31:13 +00:00

kern_fail.c

…

kern_ffclock.c

…

kern_fork.c

Reduce contention on the proctree lock during heavy package build.

2018-02-20 02:18:30 +00:00

kern_hhook.c

…

kern_idle.c

…

kern_intr.c

intr unbreak KTR/LINT build

2018-05-19 07:04:43 +00:00

kern_jail.c

Make it easier for filesystems to count themselves as jail-enabled,

2018-05-04 20:54:27 +00:00

kern_khelp.c

…

kern_kthread.c

Implement several enhancements to NUMA policies.

2018-03-29 02:54:50 +00:00

kern_ktr.c

…

kern_ktrace.c

Reduce overhead of ktrace checks in the common case.

2018-05-09 00:00:47 +00:00

kern_linker.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

kern_lock.c

remove unused locked variable in lockmgr_unlock_fast_path

2018-05-19 03:58:40 +00:00

kern_lockf.c

lockf: annotate LOCKF_DEBUG only var

2018-05-19 05:04:38 +00:00

kern_lockstat.c

lockstat: track lockstat just like sdt probes

2018-04-24 01:04:10 +00:00

kern_loginclass.c

call racct_proc_ucred_changed() under the proc lock

2018-04-20 13:08:04 +00:00

kern_malloc.c

malloc: avoid possibly returning stack garbage if MALLOC_DEBUG is defined

2018-05-19 04:43:49 +00:00

kern_mbuf.c

Add an mbuf allocator for netdump.

2018-05-06 00:19:48 +00:00

kern_mib.c

clk: Put the sysctls under hw.clock instead of clock

2018-04-27 00:12:00 +00:00

kern_module.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

kern_mtxpool.c

…

kern_mutex.c

Drop KTR_CONTENTION.

2018-03-20 15:51:05 +00:00

kern_ntptime.c

Fix several leaks of kernel stack data through paddings.

2018-03-27 18:05:51 +00:00

kern_osd.c

…

kern_physio.c

physio: avoid uninitialized variables

2018-05-19 04:09:58 +00:00

kern_pmc.c

hwpmc: fix load/unload race and vm map LOR

2018-05-14 00:21:04 +00:00

kern_poll.c

…

kern_priv.c

…

kern_proc.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

kern_procctl.c

Rename PROC_PDEATHSIG_SET -> PROC_PDEATHSIG_CTL and PROC_PDEATHSIG_GET

2018-04-20 15:19:27 +00:00

kern_prot.c

call racct_proc_ucred_changed() under the proc lock

2018-04-20 13:08:04 +00:00

kern_racct.c

call racct_proc_ucred_changed() under the proc lock

2018-04-20 13:08:04 +00:00

kern_rangelock.c

…

kern_rctl.c

call racct_proc_ucred_changed() under the proc lock

2018-04-20 13:08:04 +00:00

kern_resource.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

kern_rmlock.c

rmlock: partially depessimize lock/unlock fastpath

2018-05-11 06:59:54 +00:00

kern_rwlock.c

rw: decrease writer starvation

2018-05-22 07:16:39 +00:00

kern_sdt.c

…

kern_sema.c

…

kern_sendfile.c

sendfile: annotate unused value and ensure that npages is actually initialized

2018-05-19 05:10:51 +00:00

kern_sharedpage.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

kern_shutdown.c

remove unused variable

2018-05-19 03:55:42 +00:00

kern_sig.c

signotify: don't create a stack local that isn't used on non-debug builds

2018-05-19 03:57:41 +00:00

kern_switch.c

…

kern_sx.c

sx: port over writer starvation prevention measures from rwlock

2018-05-22 07:20:22 +00:00

kern_synch.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

kern_syscalls.c

Avoid calls to syscall_thread_enter/exit for statically defined syscalls

2018-05-07 22:29:32 +00:00

kern_sysctl.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

kern_tc.c

tc: bcopy -> memcpy

2018-05-04 22:48:10 +00:00

kern_thr.c

hwpmc: Implement per-thread counters for PMC sampling

2018-05-16 22:29:20 +00:00

kern_thread.c

hwpmc: Implement per-thread counters for PMC sampling

2018-05-16 22:29:20 +00:00

kern_time.c

…

kern_timeout.c

…

kern_tslog.c

…

kern_umtx.c

umtx: don't call umtxq_getchain unless the value is needed

2018-05-19 05:09:10 +00:00

kern_uuid.c

…

kern_xxx.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

ksched.c

…

link_elf_obj.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

link_elf.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

linker_if.m

…

Make.tags.inc

…

Makefile

…

makesyscalls.sh

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

md4c.c

…

md5c.c

…

msi_if.m

…

p1003_1b.c

…

pic_if.m

…

posix4_mib.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

sched_4bsd.c

Inlined sched_userret.

2018-05-07 23:36:16 +00:00

sched_ule.c

Inlined sched_userret.

2018-05-07 23:36:16 +00:00

serdev_if.m

…

stack_protector.c

…

subr_acl_nfs4.c

…

subr_acl_posix1e.c

…

subr_autoconf.c

…

subr_blist.c

…

subr_bufring.c

…

subr_bus_dma.c

…

subr_bus.c

Add device_quiet_children() and device_has_quiet_children()

2018-05-07 21:09:08 +00:00

subr_busdma_bufalloc.c

…

subr_capability.c

Add additional preinitialized cap_rights

2018-05-20 05:13:12 +00:00

subr_clock.c

…

subr_compressor.c

…

subr_counter.c

…

subr_devmap.c

…

subr_devstat.c

…

subr_disk.c

…

subr_dummy_vdso_tc.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

subr_epoch.c

Don't pass a section cookie to CK for non-preemptible epoch sections.

2018-05-21 16:03:51 +00:00

subr_eventhandler.c

Fix initialization of eventhandler mutex.

2018-03-19 22:43:27 +00:00

subr_fattime.c

…

subr_firmware.c

…

subr_gtaskqueue.c

Add taskqgroup_config_gtask_deinit to support teardown after

2018-05-09 18:51:35 +00:00

subr_hash.c

…

subr_hints.c

…

subr_intr.c

…

subr_kdb.c

…

subr_kobj.c

…

subr_lock.c

locks: tweak backoff a little bit

2018-04-08 16:34:10 +00:00

subr_log.c

…

subr_mchain.c

…

subr_module.c

…

subr_msgbuf.c

…

subr_param.c

…

subr_pcpu.c

Use NULL for SYSINIT's last arg, which is a pointer type

2018-05-18 17:58:09 +00:00

subr_pctrie.c

…

subr_pidctrl.c

pidctrl Actually use the variables that we assign to as seatbelts to prevent divide

2018-05-19 02:17:18 +00:00

subr_power.c

…

subr_prf.c

fix uninitialized variable warning

2018-05-19 03:49:36 +00:00

subr_prof.c

Stop using fuswintr() and suswintr() in the profiler.

2018-04-17 16:36:53 +00:00

subr_rman.c

…

subr_rtc.c

…

subr_sbuf.c

…

subr_scanf.c

…

subr_sfbuf.c

…

subr_sglist.c

…

subr_sleepqueue.c

turnstile / sleepqueue: annotate variables only used by debug builds

2018-05-19 05:00:16 +00:00

subr_smp.c

…

subr_stack.c

…

subr_syscall.c

systrace: track it like sdt probes

2018-04-27 15:16:34 +00:00

subr_taskqueue.c

…

subr_terminal.c

teken, vt(4): New callbacks to lock the terminal once

2018-05-16 09:01:02 +00:00

subr_trap.c

sx: port over writer starvation prevention measures from rwlock

2018-05-22 07:20:22 +00:00

subr_turnstile.c

turnstile / sleepqueue: annotate variables only used by debug builds

2018-05-19 05:00:16 +00:00

subr_uio.c

simplify control flow so that gcc knows we never pass save to curthread_pflags_restore

2018-05-19 04:04:44 +00:00

subr_unit.c

…

subr_vmem.c

…

subr_witness.c

ifnet: Replace if_addr_lock rwlock with epoch + mutex

2018-05-18 20:13:34 +00:00

sys_capability.c

capsicum: propagate const correctness

2018-05-19 05:14:05 +00:00

sys_generic.c

Fix the build after r333457

2018-05-10 13:19:42 +00:00

sys_getrandom.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

sys_pipe.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

sys_procdesc.c

Eliminate the overhead of gratuitous repeated reinitialization of cap_rights

2018-05-09 18:47:24 +00:00

sys_process.c

sys_process.c fix set but not used warning

2018-05-19 03:48:35 +00:00

sys_socket.c

…

syscalls.c

Regenerate sysent files after r331279.

2018-03-21 01:17:01 +00:00

syscalls.master

Added SAL annotatations to system calls.

2018-04-05 20:31:45 +00:00

systrace_args.c

Regenerate sysent files after r331279.

2018-03-21 01:17:01 +00:00

sysv_ipc.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

sysv_msg.c

sysv_msg initialize saved_msgsz

2018-05-19 03:56:39 +00:00

sysv_sem.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

sysv_shm.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

tty_compat.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

tty_info.c

…

tty_inq.c

tty: use __unused annotation instead to silence warnings

2018-05-19 04:48:26 +00:00

tty_outq.c

tty: use __unused annotation instead to silence warnings

2018-05-19 04:48:26 +00:00

tty_pts.c

Move uio enums to sys/_uio.h.

2018-03-27 15:20:03 +00:00

tty_tty.c

…

tty_ttydisc.c

…

tty.c

Move most of the contents of opt_compat.h to opt_global.h.

2018-04-06 17:35:35 +00:00

uipc_accf.c

…

uipc_debug.c

Revert r332894 at the request of the submitter.

2018-04-24 19:55:12 +00:00

uipc_domain.c

…

uipc_mbuf2.c

…

uipc_mbuf.c

…

uipc_mbufhash.c

…

uipc_mqueue.c

mqueue: avoid unused variables

2018-05-19 04:10:53 +00:00

uipc_sem.c

Eliminate the overhead of gratuitous repeated reinitialization of cap_rights

2018-05-09 18:47:24 +00:00

uipc_shm.c

…

uipc_sockbuf.c

fix uninitialized variable warning

2018-05-19 03:49:36 +00:00

uipc_socket.c

Revert r332894 at the request of the submitter.

2018-04-24 19:55:12 +00:00

uipc_syscalls.c

Eliminate the overhead of gratuitous repeated reinitialization of cap_rights

2018-05-09 18:47:24 +00:00

uipc_usrreq.c

AF_UNIX gc unused label

2018-05-20 21:37:34 +00:00

vfs_acl.c

…

vfs_aio.c

Eliminate the overhead of gratuitous repeated reinitialization of cap_rights

2018-05-09 18:47:24 +00:00

vfs_bio.c

vfs: annotate variables only used by debug builds as __unused

2018-05-19 04:59:39 +00:00

vfs_cache.c

cache_lookup remove unused variable and initialize used

2018-05-19 04:08:11 +00:00

vfs_cluster.c

Detect and optimize reads from the hole on UFS.

2018-05-13 09:47:28 +00:00

vfs_default.c

vfs: simplify vop_stdlock/unlock

2018-05-20 04:45:05 +00:00

vfs_export.c

…

vfs_extattr.c

…

vfs_hash.c

…

vfs_init.c

Make it easier for filesystems to count themselves as jail-enabled,

2018-05-04 20:54:27 +00:00

vfs_lookup.c

vfs: annotate variables only used by debug builds as __unused

2018-05-19 04:59:39 +00:00

vfs_mount.c

Make it easier for filesystems to count themselves as jail-enabled,

2018-05-04 20:54:27 +00:00

vfs_mountroot.c

Remove Giant from init creation and vfs_mountroot.

2018-03-21 14:46:54 +00:00

vfs_subr.c

vfs: annotate variables only used by debug builds as __unused

2018-05-19 04:59:39 +00:00

vfs_syscalls.c

Add additional preinitialized cap_rights

2018-05-20 05:13:12 +00:00

vfs_vnops.c

In vn_io_fault1(), reduce the scope where pagefaults are disabled.

2018-03-24 13:13:52 +00:00

vnode_if.src

…