The acquisition and release of an uncontended default/normal pthread
mutex on FreeBSD is suprisingly slow, e.g., pthread wrlocks and binary
semaphores both exhibit roughly 33% lower latency, while default/normal
mutexes on Linux exhibit roughly 67% lower latency than FreeBSD. This is
likely explained by the fact that AFAICT in the best case to acquire an
uncontended mutex on Linux one need touch only 1 page and read+modify
only 1 cacheline, whereas on FreeBSD we need to touch at least 4 pages,
read 6 cachelines, and modify at least 4 cachelines.
This patch does not address the pthread mutex architecture. Instead,
it improves performance by adding the __always_inline attribute to
mutex_lock_common() and mutex_unlock_common() to encourage constant
folding and propagation, thereby lowering the latency to acquire and
release a mutex due to a shorter code path with fewer compares, jumps,
and mispredicts.
With this patch on a stock build I see a reduction in latency of roughly
7% for default/normal mutexes, and 17% for robust mutexes. When built
without PTHREADS_ASSERTIONS enabled I see a reduction in latency of
roughly 15% and 26%, respectively. Suprisingly, I see similar reductions
in latency for heavily contended mutexes.
By default, this patch increases the size of libthr.so.3 by 2448 bytes,
but when built without PTHREAD_ASSERTIONS enabled it only increases by
448 bytes.
Reviewed by: jhb (previous version), kib
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40912
This patch fixes a bug which prevents building libthr without
_PTHREADS_INVARIANTS defined. The default remains to build libthr
with -D_PTHREADS_INVARIANTS. However, with this patch, if one builds
libthr with WITHOUT_PTHREADS_ASSERTIONS=true then the latency to
acquire+release a default pthread mutex is reduced by roughly 5%, and a
robust mutex by roughly 18% (as measured by a simple synthetic test on a
Xeon E5-2697a based machine).
Reviewed by: jhb, kib, mjg
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40900
Reduces severe performance degradation due to false-sharing. Note that this
does not account for hardware which can perform adjacent cacheline prefetch.
[mjg: massaged the commit message and the patch to use aligned_alloc
instead of malloc]
PR: 272238
MFC after: 1 week
Since there is only the current thread in the child, no pending readers
exist. Clear the bit, since it confuses future attempts to acquire
write ownership of the rtld locks, due to URWLOCK_PREFER_READERS flag.
To be future-proof, clear all state about pending writers and readers.
PR: 271490
Reported and tested by: KJ Tsanaktsidis <kj@kjtsanaktsidis.id.au>
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D40178
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
When __thr_pshared_offpage() is called for allocation, it must not use
the cached offpage for the key. Instead, the cached offpage must be
unmapped and removed from the cache, if any.
It is legitimate for the user code to unmap the shared lock object without
destroying it, and then mapping something over the freed VA to carry
another shared lock. In this case the cached offpage must be un-cached.
PR: 269277
Reported by: rau8344@gmail.com
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D38345
Since f35093f8 semantics of a thread affinity functions is changed to be a
compatible with Linux:
In case of getaffinity(), the minimum cpuset_t size that the kernel permits is
the maximum CPU id, present in the system, / NBBY bytes, the maximum size is not
limited.
In case of setaffinity(), the kernel does not limit the size of the user-provided
cpuset_t, internally using only the meaningful part of the set, where the upper
bound is the maximum CPU id, present in the system, no larger than the size of
the kernel cpuset_t.
To match pthread_attr_[g|s]etaffinity_np checks of the user-provided cpusets to
the kernel behavior export the minimum cpuset_t size allowed by running kernel
via new sysctl kern.sched.cpusetsizemin and use it in checks.
Reviewed by:
Differential Revision: https://reviews.freebsd.org/D38112
MFC after: 1 week
In libthr we use PAGE_SIZE when allocating memory with mmap and to check
various structs will fit into a single page so we can use this allocator
for them.
Ask the kernel for the page size on init for use by the page allcator
and add a new machine dependent macro to hold the smallest page size
the architecture supports to check the structure is small enough.
This allows us to use the same libthr on arm64 with either 4k or 16k
pages.
Reviewed by: kib, markj, imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34984
Rather than calling getpagesize() twice use the value saved after the
first call to size a mmap allocation.
Reviewed by: kib, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34983
When a thread exits, _Unwind_ForcedUnwind() is used to walk up stack
frames executing pending cleanups pushed by pthread_cleanup_push().
The cleanups are popped by thread_unwind_stop() which is passed as a
callback function to _Unwind_ForcedUnwind().
LLVM's libunwind uses a different function type for the callback on
32-bit ARM relative to all other platforms. The previous unwind.h
header (as well as the unwind.h from libcxxrt) use the non-ARM type on
all platforms, so this has likely been broken on 32-bit arm since it
switched to using LLVM's libunwind.
For now, just disable stack unwinding on 32-bit arm to unbreak the
build until a proper fix is tested.
Install headers from LLVM's libunwind in place of the headers from
libcxxrt and allow C applications to use the library.
As part of this, remove include/unwind.h and switch libthr over to
using the installed unwind.h.
Reviewed by: dim, emaste
MFC after: 10 days
Differential Revision: https://reviews.freebsd.org/D34065
This matches the type in other unwind headers (LLVM libunwind,
libcxxrt, glibc).
NB: include/unwind.h is not installed but is only used by libthr
Reviewed by: imp, dim, emaste
Differential Revision: https://reviews.freebsd.org/D34049
This matches libc and rtld in using the alignment (TLS_TCB_ALIGN) from
machine/tls.h instead of hardcoding 16.
Reviewed by: kib
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D34023
The current ASLR stack gap feature will be removed, and with that the
need for this change, and the kern.stactop sysctl, is gone. Moreover,
the approach taken in this revision does not provide compatibility for
old copies of libthr.so, and the revision should have also updated
__libc_map_stacks_exec().
This reverts commit 78df56ccfc.
Reviewed by: kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33704
Note that on amd64 this effectively removes the unused tcb_spare field
from the end of struct tcb since the definition of struct tcb in
<x86/tls.h> does not include that field.
Reviewed by: kib, jrtc27
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33352
This macro is confusing as it is not related to the similarly named
TLS_DTV_OFFSET. Instead, replace its one use with the desired
expression which is the same on all platforms.
Reviewed by: kib, emaste, jrtc27
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33345
Use the new kern.stacktop sysctl to retrieve the address of stack top
instead of kern.usrstack. kern.usrstack does not have any knowledge
of the stack gap, so this can cause problems with thread stacks.
Using kern.stacktop sysctl should fix most of those problems.
kern.usrstack is used as a fallback when kern.stacktop cannot be read.
Rename usrstack variables to stacktop to reflect this change.
Fixes problems with firefox and thunderbird not starting with
stack gap enabled.
PR: 239873
Reviewed by: kib
Obtained from: Semihalf
Sponsored by: Stormshield
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D31898
Describe internal allocations, mention problems with the use of global
malloc(3) and the reasons for internal allocator existence.
Document shared objects implementation and describe shortcomings of the
chosen approach, as well as the rationale why it was done that way.
Reviewed by: markj
Discussed with: jilles
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Differential revision: https://reviews.freebsd.org/D32243
I got the following error with an ASAN-instrument libthr:
==803==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffffffcdb0 at pc 0x000801863396 bp 0x7ff8
READ of size 4 at 0x7fffffffcdb0 thread T0
#0 0x801863395 in handle_signal /local/scratch/alr48/cheri/freebsd/lib/libthr/thread/thr_sig.c:262:2
#1 0x801860da2 in thr_sighandler /local/scratch/alr48/cheri/freebsd/lib/libthr/thread/thr_sig.c:246:2
Address 0x7fffffffcdb0 is located in stack of thread T0 at offset 208 in frame
#0 0x80186080f in thr_sighandler /local/scratch/alr48/cheri/freebsd/lib/libthr/thread/thr_sig.c:213
This frame has 1 object(s):
[32, 64) 'act' (line 216) <== Memory access at offset 208 overflows this variable
HINT: this may be a false positive if your program uses some custom stack
This seems like a false-positive since the line in question is
`SIGSETOR(actp->sa_mask, ucp->uc_sigmask);` and it complains about a read
operation (from the ucontext_t argument) so this indicates to me that ASAN
does not understand that thr_sighandler() is a signal handler.
Differential Revision: https://reviews.freebsd.org/D31074
Right now, libthr does not initialize RtldLockInfo.rtli_version when calling
_rtld_thread_init(), which makes versioning the interface troublesome.
Add a workaround: if the calling object of _rtld_thread_init() exports
the "_pli_rtli_version" symbol, then consider rtli_version initialized.
Otherwise, forcibly set it to RTLI_VERSION_ONE, currently defined as
RTLI_VERSION.
Export "_pli_rtli_version" from libthr and properly initialize rtli_version.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D29633
This was only needed on 32-bit arm prior to ARMv6. As we only support
ARMv6 or later remove it.
Reviewed by: mannu
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D29624
This warning is very rarely useful (inline is a hint and not mandatory).
This flag results in many warnings being printed when compiling C++
code that uses the standard library with GCC.
This flag was originally added in back in r94332 but the flag is a no-op
in Clang ("This diagnostic flag exists for GCC compatibility, and has no
effect in Clang"). Removing it should make the GCC build output slightly
more readable.
Reviewed By: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D29235
One possible way the recursion can happen is during fork: suppose
that fork is called from early code that did not triggered
jemalloc(3) initialization yet. Then we lock thr_malloc lock, and
call malloc_prefork() that might require initialization of jemalloc
pthread_mutexes, calling into libthr malloc. It is safe to allow
recursion for this occurence.
PR: 252579
Reported by: Vasily Postnicov <shamaz.mazum@gmail.com>
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Without wrapping, rtld services and malloc(3) are not guaranteed
to operate correctly in the forked child.
Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28088
Follow-up to r353959 and r368070: do the same for other architectures.
arm32 already seems to use its own .fnstart/.fnend directives, which
appear to be ARM-specific variants of the same thing. Likewise, MIPS
uses .frame directives.
Reviewed by: arichardson
Differential Revision: https://reviews.freebsd.org/D27387
We need at least thr_malloc ready. The situation is possible e.g. in case
of libthr being listed in DT_NEEDED before some of its consumers.
Reported and tested by: lev
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
This seems to be required by recent clang asan.
I do not see other way than put the symbol under FBSD_1.0 version.
PR: 251112
Reported by: Andrew Stitcher <astitcher@apache.org>
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27389
Specifically, if we're waking up some value n > BATCH_SIZE, then the
copyin(9) is wrong on the second iteration due to upp being the wrong type.
upp is currently a uint32_t**, so upp + pos advances it by twice as many
elements as it should (host pointer size vs. compat32 pointer size).
Fix it by just making upp a uint32_t*; it's still technically a double
pointer, but the distinction doesn't matter all that much here since we're
just doing arithmetic on it.
Add a test case that demonstrates the problem, placed with the libthr tests
since one messing with _umtx_op should be running these tests. Running under
compat32, the new test case will hang as threads after the first 128 get
missed in the wake. it's not immediately clear how to hit it in practice,
since pthread_cond_broadcast() uses a smaller (sleepq batch?) size observed
to be around ~50 -- I did not spend much time digging into it.
The uintptr_t change makes no functional difference, but i've tossed it in
since it's more accurate (semantically).
Reported by: Andrew Gierth (andrew_tao173.riddles.org.uk, inspection)
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D27231
The current default is provided in various Makefile.inc in some top-level
directories and covers a good portion of the tree, but doesn't cover parts
of the build a little deeper (e.g. libcasper).
Provide a default in src.sys.mk and set WARNS to it in bsd.sys.mk if that
variable is defined. This lets us relatively cleanly provide a default WARNS
no matter where you're building in the src tree without breaking things
outside of the tree.
Crunchgen has been updated as a bootstrap tool to work on this change
because it needs r365605 at a minimum to succeed. The cleanup necessary to
successfully walk over this change on WITHOUT_CLEAN builds has been added.
There is a supplemental project to this to list all of the warnings that are
encountered when the environment has WARNS=6 NO_WERROR=yes:
https://warns.kevans.dev -- this project will hopefully eventually go away
in favor of CI doing a much better job than it.
Reviewed by: emaste, brooks, ngie (all earlier version)
Reviewed by: emaste, arichardson (depend-cleanup.sh change)
Differential Revision: https://reviews.freebsd.org/D26455
pthread_get_name_np() and pthread_set_name_np().
This re-applies r361770 after compatibility fixes.
Reviewed by: antoine, jkim, markj
Tested by: antoine (exp-run)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D25117
