Commit Graph

15299 Commits

Author SHA1 Message Date
Emmanuel Vadot
beaa6e1e64 subr_sfbus.c need sys/proc.h for struct thread definition.
This fixes kernel build for armv6.

Discussed with: kib
2017-02-07 17:31:24 +00:00
Mateusz Guzik
dbccc8105c rwlock: implement RW_LOCK_WRITER_RECURSED bit
This moves recursion handling out of the inlined wunlock path and in
particular saves a read and a branch.

Discussed with:
2017-02-07 17:04:31 +00:00
Mateusz Guzik
f743ea9638 Bump struct thread alignment to 32.
This gives additional bits to use in locking primitives which store
the lock thread pointer in the lock value.

Discussed with:	kib
2017-02-07 17:03:22 +00:00
Mateusz Guzik
3c798b2b1f locks: follow up r313386
Unfinished diff was committed by accident. The loop in lock_delay
was changed to decrement, but the loop iterator was still incrementing.
2017-02-07 16:01:07 +00:00
Mateusz Guzik
8e5a3e9a9d locks: change backoff to exponential
Previous implementation would use a random factor to spread readers and
reduce chances of starvation. This visibly reduces effectiveness of the
mechanism.

Switch to the more traditional exponential variant. Try to limit starvation
by imposing an upper limit of spins after which spinning is half of what
other threads get. Note the mechanism is turned off by default.

Reviewed by:	kib (previous version)
2017-02-07 14:49:36 +00:00
Edward Tomasz Napierala
1110d0029a Make root_mount_hold() work after boot. This is important for two
reasons. First is rerooting into USB-mounted device that happens
to be not yet enumerated. The second is when mounting with (non-root)
filesystem on USB device on a hub that's enumerated later than the root
mount: the rc scripts explicitly mount for the root mount holds to be
released, but each USB bus takes the hold asynchronously, and if that
happens after root mount, it would just get ignored.

Reviewed by:	marcel
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9388
2017-02-06 20:44:34 +00:00
Edward Tomasz Napierala
4f9d7bad48 In r290196 the root mount hold mechanism was changed to make it not wait
for mount hold release if the root device already exists.  So, unless your
rootdev is not on USB - ie in the usual case - the root mount won't wait
for USB.  However, the old behaviour was sometimes used as "wait until USB
is fully enumerated", and r290196 broke that.

This commit adds vfs.root_mount_always_wait tunable, to force the kernel
to always wait for root mount holds, even if the root is already there.

Reviewed by:	kib
MFC after:	2 weeks
Relnotes:	yes
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9387
2017-02-06 20:36:59 +00:00
Andrew Turner
c0d5237034 Only allow the pic type to be either a PIC or MSI type. All interrupt
controller drivers handle either MSI/MSI-X interrupts, or regular
interrupts, as such enforce this in the interrupt handling framework.
If a later driver was to handle both it would need to create one of each.

This will allow future changes to allow the xref space to overlap, but
refer to different drivers.

Obtained from:	ABT Systems Ltd
Sponsored by:	The FreeBSD Foundation
X-Differential Revision:	https://reviews.freebsd.org/D8616
2017-02-06 13:08:48 +00:00
Mateusz Guzik
c1aaf63cb5 locks: fix recursion support after recent changes
When a relevant lockstat probe is enabled the fallback primitive is called with
a constant signifying a free lock. This works fine for typical cases but breaks
with recursion, since it checks if the passed value is that of the executing
thread.

Read the value if necessary.
2017-02-06 09:40:14 +00:00
Mateusz Guzik
993ddec44d rwlock: move lockstat handling out of inline primitives
See r313275 for details.

One difference here is that recursion handling was removed from the fallback
routine. As it is it was never supposed to see a recursed lock in the first
place. Future changes will move it out of inline variants, but right now
there is no easy to way to test if the lock is recursed without reading
additional words.
2017-02-05 13:37:23 +00:00
Edward Tomasz Napierala
96ee43103d Add kern_cpuset_getaffinity() and kern_cpuset_getaffinity(),
and use it in compats instead of their sys_*() counterparts.

Reviewed by:	kib, jhb, dchagin
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9383
2017-02-05 13:24:54 +00:00
Mateusz Guzik
6ebb77b6a6 sx: move lockstat handling out of inline primitives
See r313275 for details.
2017-02-05 09:54:16 +00:00
Mateusz Guzik
dc0896512c mtx: fixup r313278, the assignemnt was supposed to go inside the loop 2017-02-05 09:53:13 +00:00
Mateusz Guzik
cae4ab7f37 mtx: fix up _mtx_obtain_lock_fetch usage in thread lock
Since _mtx_obtain_lock_fetch no longer sets the argument to MTX_UNOWNED,
callers have to do it on their own.
2017-02-05 09:35:17 +00:00
Mateusz Guzik
08da267775 mtx: move lockstat handling out of inline primitives
Lockstat requires checking if it is enabled and if so, calling a 6 argument
function. Further, determining whether to call it on unlock requires
pre-reading the lock value.

This is problematic in at least 3 ways:
- more branches in the hot path than necessary
- additional cacheline ping pong under contention
- bigger code

Instead, check first if lockstat handling is necessary and if so, just fall
back to regular locking routines. For this purpose a new macro is introduced
(LOCKSTAT_PROFILE_ENABLED).

LOCK_PROFILING uninlines all primitives. Fold in the current inline lock
variant into the _mtx_lock_flags to retain the support. With this change
the inline variants are not used when LOCK_PROFILING is defined and thus
can ignore its existence.

This results in:
   text	   data	    bss	    dec	    hex	filename
22259667	1303208	4994976	28557851	1b3c21b	kernel.orig
21797315	1303208	4994976	28095499	1acb40b	kernel.patched

i.e. about 3% reduction in text size.

A remaining action is to remove spurious arguments for internal kernel
consumers.
2017-02-05 08:04:11 +00:00
Mateusz Guzik
3ae56ce958 sx: add witness support missed in r313272 2017-02-05 06:51:45 +00:00
Mateusz Guzik
9d2e4290ff sx: uninline slock/sunlock
Shared locking routines explicitly read the value and test it. If the
change attempt fails, they fall back to a regular function which would
retry in a loop.

The problem is that with many concurrent readers the risk of failure is pretty
high and even the value returned by fcmpset is very likely going to be stale
by the time the loop in the fallback routine is reached.

Uninline said primitives. It gives a throughput increase when doing concurrent
slocks/sunlocks with 80 hardware threads from ~50 mln/s to ~56 mln/s.

Interestingly, rwlock primitives are already not inlined.
2017-02-05 05:20:29 +00:00
Mateusz Guzik
fa47404353 sx: switch to fcmpset
Discussed with:	jhb
Tested by:	pho (previous version)
2017-02-05 04:54:20 +00:00
Mateusz Guzik
c84f347985 rwlock: switch to fcmpset
Discussed with:	jhb
Tested by:	pho
2017-02-05 04:53:13 +00:00
Mateusz Guzik
90836c3270 mtx: switch to fcmpset
The found value is passed to locking routines in order to reduce cacheline
accesses.

mtx_unlock grows an explicit check for regular unlock. On ll/sc architectures
the routine can fail even if the lock could have been handled by the inline
primitive.

Discussed with:	jhb
Tested by:	pho (previous version)
2017-02-05 03:26:34 +00:00
Mateusz Guzik
2d78a5531e vfs: use atomic_fcmpset in vfs_refcount_* 2017-02-05 03:23:16 +00:00
Mark Johnston
69d2418faa Make witness_warn() always print to the console.
witness_warn() either breaks into the debugger or panics the system, so its
output should go to the console regardless of the witness(4) output channel
configuration.

MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-02-05 02:27:04 +00:00
Mateusz Guzik
3a2f282532 fd: switch fget_unlocked to atomic_fcmpset 2017-02-05 01:40:27 +00:00
Jason A. Harmening
ad62ba6e96 Revert r313037
The switch to get_pcpu() in MI code seems to cause hangs on MIPS.
Back out until we can get a better idea of what's happening there.

Reported by:	kan, lidl
2017-02-04 06:24:49 +00:00
Hartmut Brandt
4b481ba0ed Merge filt_soread and filt_solisten and decide what to do when checking
for EVFILT_READ at the point of the check not when the event is registers.
This fixes a problem with asio when accepting a connection.

Reviewed by:	kib@, Scott Mitchell
2017-02-01 13:12:07 +00:00
Jason A. Harmening
65ed483615 Implement get_pcpu() for the remaining architectures and use it to
replace pcpu_find(curcpu) in MI code.
2017-02-01 03:32:49 +00:00
Edward Tomasz Napierala
b38b22b0b2 Add kern_pread() and kern_pwrite(), and use it in compats instead
of their sys_*() counterparts. The svr4 is left unchanged.

Reviewed by:	kib@
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9379
2017-01-31 15:35:18 +00:00
Edward Tomasz Napierala
fc8bde8ffe Replace calls to sys_truncate() with kern_truncate().
Reviewed by:	kib@
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9371
2017-01-31 15:19:44 +00:00
Edward Tomasz Napierala
ea2ebdc19e Add kern_cpuset_getid() and kern_cpuset_setid(), and use them
in compat32 instead of their sub_*() counterparts.

Reviewed by:	jhb@, kib@
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9382
2017-01-31 15:11:23 +00:00
Andriy Gapon
826b3d3187 put very expensive sanity checks of advisory locks under DIAGNOSTIC
The checks have quadratic complexity over a number of advisory locks
active for a file and that could be a lot.  What's the worse is that the
checks are done while holding ls_lock.  That could lead to a long a very
long backlog and performance degradation even if all requested locks are
compatible (e.g. all shared locks).

The checks used to be under INVARIANTS.

Discussed with:	kib
MFC after:	2 weeks
Sponsored by:	Panzura
2017-01-30 15:20:13 +00:00
Edward Tomasz Napierala
d293f35c09 Add kern_listen(), kern_shutdown(), and kern_socket(), and use them
instead of their sys_*() counterparts in various compats. The svr4
is left untouched, because there's no point.

Reviewed by:	ed@, kib@
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9367
2017-01-30 12:57:22 +00:00
Edward Tomasz Napierala
f67d6b5f12 Add kern_lseek() and use it instead of sys_lseek() in various compats.
I didn't touch svr4/, there's no point.

Reviewed by:	ed@, kib@
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9366
2017-01-30 12:24:47 +00:00
Edward Tomasz Napierala
ae6b6ef6cb Replace sys_ftruncate() with kern_ftruncate() in various compats.
Reviewed by:	kib@
MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D9368
2017-01-30 11:50:54 +00:00
Mateusz Guzik
dfecf51dd0 cache: use vrefact for '.' lookups and refing the rdir in fullpath 2017-01-30 03:20:05 +00:00
Mateusz Guzik
3071469d57 fd: sprinkle __read_mostly and __exclusive_cache_line 2017-01-30 03:07:32 +00:00
Baptiste Daroussin
b4b4b5304b Revert crap accidentally committed 2017-01-28 16:31:23 +00:00
Baptiste Daroussin
814aaaa7da Revert r312923 a better approach will be taken later 2017-01-28 16:30:14 +00:00
Mateusz Guzik
95839d3d25 hwpmc: annotate pmc_hook and pmc_intr as __read_mostly
MFC after:	1 month
2017-01-27 22:14:42 +00:00
Mateusz Guzik
f1f7f1cb29 hwpmc: partially depessimize mmap handling if the module is not loaded
In particular this means the pmc sx lock is no longer taken when an
executable mapping succeeds.

MFC after:	1 week
2017-01-27 22:13:15 +00:00
Mateusz Guzik
290511163d Sprinkle __read_mostly on backoff and lock profiling code.
MFC after:	1 month
2017-01-27 15:03:51 +00:00
Mateusz Guzik
17071ff298 cache: annotate with __read_mostly and __exclusive_cache_line
MFC after:	1 month
2017-01-27 14:56:36 +00:00
Sean Bruno
de414cfe14 A few more style bugs lying around in here.
Submitted by:	bde
2017-01-26 13:48:45 +00:00
Gleb Smirnoff
beb4b31200 For non-listening AF_UNIX sockets return error code EOPNOTSUPP to match
documentation and SUS.
2017-01-25 22:26:45 +00:00
Ed Maste
f27ac8e297 ANSIfy kern_ntptime.c 2017-01-25 20:22:32 +00:00
Sean Bruno
06bb7c507a Replace overlooked smp_started checks and variable use in a print
with the now used tqg_smp_started.

Submitted by:	bde
2017-01-25 15:54:44 +00:00
Ed Maste
77ebe276ba imgact_elf: refactor et_dyn_addr calculation
This simplifies the logic somewhat. It is extracted from the change in
review in D5603.

Differential Revision:	https://reviews.freebsd.org/D9321
2017-01-24 22:46:43 +00:00
Mateusz Guzik
543b2f425d proc: perform a lockless check in sys_issetugid
Discussed with:	kib
MFC after:	1 week
2017-01-24 21:48:57 +00:00
Conrad Meyer
90a79ac576 Use time_t for intermediate values to avoid overflow in clock_ts_to_ct
Add additionally safety and overflow checks to clock_ts_to_ct and the
BCD routines while we're here.

Perform a safety check in sys_clock_settime() first to avoid easy local
root panic, without having to propagate an error value back through
dozens of APIs currently lacking error returns.

PR:		211960, 214300
Submitted by:	Justin McOmie <justin.mcomie at gmail.com>, kib@
Reported by:	Tim Newsham <tim.newsham at nccgroup.trust>
Reviewed by:	kib@
Sponsored by:	Dell EMC Isilon, FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D9279
2017-01-24 18:05:29 +00:00
Sean Bruno
bd84f70044 iflib:
Add internal tracking of smp startup status to reliably figure out
     what methods are to be used to get gtaskqueue up and running.

e1000:
     Calculating this pointer gives undefined behaviour when (last == -1)
     (it is before the buffer).  The pointer is always followed.  Panics
     occurred when it points to an unmapped page.  Otherwise, the pointed-to
     garbage tends to not have the E1000_TXD_STAT_DD bit set in it, so in the
     broken case the loop was usually null and the function just returned, and
     this was acidentally correct.

Submitted by:	bde
Reported by:	Matt Macy <mmacy@nextbsd.org>
2017-01-24 16:05:42 +00:00
Sean Bruno
36fa5d5b64 Revert 312696 due to build tests. 2017-01-24 15:55:52 +00:00