Without wrapping, rtld services and malloc(3) are not guaranteed
to operate correctly in the forked child.
Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28088
Ensure proper handshake to transfer sigfastblock(2) blocking word
ownership from rtld to libthr.
Unfortunately sigfastblock(2) is not enough to stop intercepting
signals in libthr, because critical sections must ensure more than
just signal blocking.
Tested by: pho
Disscussed with: cem, emaste, jilles
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D12773
In some corner cases of static linking and unexpected libraries order
on the linker command line, libc symbol might preempt the same libthr
symbol, in which case libthr jump table points back to libc causing
either infinite recursion or loop. Handle all of such symbols by
using private libthr names for them, ensuring that the right pointers
are installed into the table.
In collaboration with: arichardson
PR: 239475
Tested by: pho
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D21088
In some corner cases of static linking and unexpected libraries order
on the linker command line, libc symbol might preempt the same libthr
symbol, in which case libthr jump table points back to libc causing
either infinite recursion or loop. Handle all of such symbols by
using private libthr names for them, ensuring that the right pointers
are installed into the table.
In collaboration with: arichardson
PR: 239475
Tested by: pho
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D21088
If dso uses initial exec TLS mode, rtld tries to allocate TLS in
static space. If there is no space left, the dlopen(3) fails. If space
if allocated, initial content from PT_TLS segment is distributed to
all threads' pcbs, which was missed and caused un-initialized TLS
segment for such dso after dlopen(3).
The mode is auto-detected either due to the relocation used, or if the
DF_STATIC_TLS dynamic flag is set. In the later case, the TLS segment
is tried to allocate earlier, which increases chance of the dlopen(3)
to succeed. LLD was recently fixed to properly emit the flag, ld.bdf
did it always.
Initial test by: dumbbell
Tested by: emaste (amd64), ian (arm)
Tested by: Gerald Aryeetey <aryeeteygerald_rogers.com> (arm64)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D19072
The need to use libc malloc(3) from some places in libthr always
caused issues. For instance, per-thread key allocation was switched to
use plain mmap(2) to get storage, because some third party mallocs
used keys for implementation of calloc(3).
Even more important, libthr calls calloc(3) during initialization of
pthread mutexes, and jemalloc uses pthread mutexes. Jemalloc provides
some way to both postpone the initialization, and to make
initialization to use specialized allocator, but this is very fragile
and often breaks. See the referenced PR for another example.
Add the small malloc implementation used by rtld, to libthr. Use it in
thr_spec.c and for mutexes initialization. This avoids the issues with
mutual dependencies between malloc and libthr in principle. The
drawback is that some more allocations are not interceptable for
alternate malloc implementations. There should be not too much memory
use from this allocator, and the alternative, direct use of mmap(2) is
obviously worse.
PR: 235211
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D18988
Also remove logic to avoid unnecessary stores to the global variable.
Thread creation and destruction are heavy enough that any supposed savings
is in the noise.
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
The function retrieves the thread name previously set by
pthread_set_name_np(3). The name is cached in the process memory.
Requested by: Willem Jan Withagen <wjw@digiware.nl>
Man page update: Yuri Pankov <yuripv@yuripv.net>
Reviewed by: ian (previous version)
Discussed with: arichardson, bjk (man page)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D16702
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using mis-identified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
Fix regression by r318539. The sysutils/pstack uses library libthread_db to
read information about threads state. The function pt_ta_new makes lookup of
several key symbols including _thread_keytable. But r318539 mades this field
static. It causes silent ignore of libthr library by pstack and as result
sysutils/pstack doesn't output any thread information.
This fix changes this field back to non-static.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D11738
Fix warnings about:
- redundant declarations
- a local variable shadowing a global function (dlinfo)
- an old-style function definition (with an empty parameter list)
- a variable that is possibly used uninitialized
"make tinderbox" passes this time, except for a few unrelated
kernel failures.
Reviewed by: kib
MFC after: 3 days
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D10870
Add a clock_nanosleep() syscall, as specified by POSIX.
Make nanosleep() a wrapper around it.
Attach the clock_nanosleep test from NetBSD. Adjust it for the
FreeBSD behavior of updating rmtp only when interrupted by a signal.
I believe this to be POSIX-compliant, since POSIX mentions the rmtp
parameter only in the paragraph about EINTR. This is also what
Linux does. (NetBSD updates rmtp unconditionally.)
Copy the whole nanosleep.2 man page from NetBSD because it is complete
and closely resembles the POSIX description. Edit, polish, and reword it
a bit, being sure to keep any relevant text from the FreeBSD page.
Reviewed by: kib, ngie, jilles
MFC after: 3 weeks
Relnotes: yes
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D10020
Replace uses of the GCC __nonnull__ attribute with the clang nullability
qualifiers. The replacement should be transparent for clang developers as
the new qualifiers will produce the same warnings and will be useful for
static checkers but will not cause aggressive optimizations.
GCC will not produce such warnings and developers will have to use
upgraded GCC ports built with the system headers from r312538.
Hinted by: Apple's Libc-1158.20.4, Bionic libc
MFC after: 11.1 Release
Differential Revision: https://reviews.freebsd.org/D9004
This avoids unneccessary casts and make the calls to _thr_ucond_*()
functions less questionable.
The c_spare field was not included into struct pthread_cond, so the
change modifies libthr ABI for shared condvars. But since an off-page
does not legitimately contains any other data past the struct
pthread_cond, the change keeps shared condvars from pre- and post-
changed libthr compatible. Also note that the whole struct ucond was
never copied in or out by kernel.
For private condvars, the privately allocated memory was never exposed
outside libthr.
Sponsored by: The FreeBSD Foundation
intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013.
A robust mutex is guaranteed to be cleared by the system upon either
thread or process owner termination while the mutex is held. The next
mutex locker is then notified about inconsistent mutex state and can
execute (or abandon) corrective actions.
The patch mostly consists of small changes here and there, adding
neccessary checks for the inconsistent and abandoned conditions into
existing paths. Additionally, the thread exit handler was extended to
iterate over the userspace-maintained list of owned robust mutexes,
unlocking and marking as terminated each of them.
The list of owned robust mutexes cannot be maintained atomically
synchronous with the mutex lock state (it is possible in kernel, but
is too expensive). Instead, for the duration of lock or unlock
operation, the current mutex is remembered in a special slot that is
also checked by the kernel at thread termination.
Kernel must be aware about the per-thread location of the heads of
robust mutex lists and the current active mutex slot. When a thread
touches a robust mutex for the first time, a new umtx op syscall is
issued which informs about location of lists heads.
The umtx sleep queues for PP and PI mutexes are split between
non-robust and robust.
Somewhat unrelated changes in the patch:
1. Style.
2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared
pi mutexes.
3. Removal of the userspace struct pthread_mutex m_owner field.
4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls
the lifetime of the shared mutex associated with a vnode' page.
Reviewed by: jilles (previous version, supposedly the objection was fixed)
Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects)
Tested by: pho
Sponsored by: The FreeBSD Foundation
is safe to call pthread_mutex_init() on the same shared mutex several
times. POSIX claims that the behaviour in this case is undefined.
Make this working by only allowing one caller to initialize the mutex.
Other callers either see already completed initialization and do
nothing, or busy-loop yielding while designated initializer finishes.
Also make the API requirements loose by initializing mutexes on other
pthread_mutex*() calls if they see uninitialized shared mutex.
Only mutexes provide the hack for now, but it could be also
implemented for other process shared primitives from libthr.
Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com>
Sponsored by: The FreeBSD Foundation
of the pshared hash in child is consistent and can be safely used.
Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com>
Sponsored by: The FreeBSD Foundation
breaking the ABI. Special value is stored in the lock pointer to
indicate shared lock, and offline page in the shared memory is
allocated to store the actual lock.
Reviewed by: vangyzen (previous version)
Discussed with: deischen, emaste, jhb, rwatson,
Martin Simmons <martin@lispworks.com>
Tested by: pho
Sponsored by: The FreeBSD Foundation
do not participate in the global symbols namespace, but rtld locks are
still replaced and functions are interposed. In particular,
__pthread_map_stacks_exec is resolved to the libc version. If a
library is loaded later, which requires adjustment of the stack
protection mode, rtld calls into libc __pthread_map_stacks_exec due to
the symbols scope. The libc version might recurse into binder and
recursively acquire rtld bind lock, causing the hang.
Make libc __pthread_map_stacks_exec() interposed, which synchronizes
rtld locks and version of the stack exec hook when libthr loaded,
regardless of the symbol scope control or symbol resolution order.
The __pthread_map_stacks_exec() symbol is removed from the private
version in libthr since libc symbol now operates correctly in presence
of libthr.
Reported and tested by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
particular, stdio locking was affected.
Reported and tested by: "Matthew D. Fuller" <fullermd@over-yonder.net>
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
The amd64, i386, and sparc64 versions were identical, with the one
difference where the former two used inline asm instead of _tcb_get. I
have compared the function before and after replacing the asm with _tcb_get
and found the object files to be identical.
The arm, mips, and powerpc versions were almost identical. The only
difference was the powerpc version used an alignment of 1 where arm and
mips used 16. As this is an increase in alignment is will be safe.
Along with this arm, mips, and powerpc all passed, when initial was true,
the value returned from _tcb_get as the first argument to
_rtld_allocate_tls. This would then return this pointer back to the caller.
We can remove these extra calls by checking if initial is set and setting
the thread control block directly. As this is what the sparc64 code does
we can use it directly.
As after these observations all the architectures can now have identical
code we can merge them into a common file.
Differential Revision: https://reviews.freebsd.org/D1556
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
cancellation-handling code in the libthr. Translate some syscalls
into their more generic counterpart, and remove translated syscalls
from the table.
List of the affected syscalls:
creat, open -> openat
raise -> thr_kill
sleep, usleep -> nanosleep
pause -> sigsuspend
wait, wait3, waitpid -> wait4
Suggested and reviewed by: jilles (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
though GOT, by staticizing and hiding. Add setter for
__error_selector to hide it as well.
Suggested and reviewed by: jilles
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
(or loading a dso linked to libthr.so into process which was not
linked against threading library).
- Remove libthr interposers of the libc functions, including
__error(). Instead, functions calls are indirected through the
interposing table, similar to how pthread stubs in libc are already
done. Libc by default points either to syscall trampolines or to
existing libc implementations. On libthr load, libthr rewrites the
pointers to the cancellable implementations already in libthr. The
interposition table is separate from pthreads stubs indirection
table to not pull pthreads stubs into static binaries.
- Postpone the malloc(3) internal mutexes initialization until libthr
is loaded. This avoids recursion between calloc(3) and static
pthread_mutex_t initialization.
- Reinstall signal handlers with wrapper on libthr load. The
_rtld_is_dlopened(3) is used to avoid useless calls to sigaction(2)
when libthr is statically referenced from the main binary.
In the process, fix openat(2), swapcontext(2) and setcontext(2)
interposing. The libc symbols were exported at different versions
than libthr interposers. Export both libc and libthr versions from
libc now, with default set to the higher version from libthr.
Remove unused and disconnected swapcontext(3) userspace implementation
from libc/gen.
No objections from: deischen
Tested by: pho, antoine (exp-run) (previous versions)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
It is automatically set when -fPIC is passed to the compiler.
Reviewed by: dim, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D1179
unlocking the rtld bind lock results in the processing of ast and
recursing into the check_deferred_signal(). Nested execution of
check_deferred_signal() delivers the signal to user code and clears
si_signo. On return, top-level check_deferred_signal() frame
continues delivering the same signal one more time, but now with zero
si_signo.
Fix this by adding a flag to indicate that deferred delivery is
running, so check_deferred_signal() should avoid doing anything. Since
user signal handler is allowed to modify the passed machine context to
make return from the signal handler to cause arbitrary jump, or do
longjmp(). For this case, also clear the flag in thr_sighandler(),
since kernel signal delivery means that nested delivery code should
not run right now.
Reported by: Vitaly Magerya <vmagerya@gmail.com>
Reviewed by: davidxu, jilles
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
pthread_suspend_all_np() may have already suspended its parent thread.
Add locking code in pthread_suspend_all_np() to only allow one thread
to suspend other threads, this eliminates a deadlock where two or more
threads try to suspend each others.
Enqueue thread in LIFO, this can cause starvation, but it gives better
performance. Use _thr_queuefifo to control the frequency of FIFO vs LIFO,
you can use environment string LIBPTHREAD_QUEUE_FIFO to configure the
variable.
a mutex after a thread has unlocked it, it event writes data to the mutex
memory to clear contention bit, there is a race that other threads
can lock it and unlock it, then destroy it, so it should not write
data to the mutex memory if there isn't any waiter.
The new operation UMTX_OP_MUTEX_WAKE2 try to fix the problem. It
requires thread library to clear the lock word entirely, then
call the WAKE2 operation to check if there is any waiter in kernel,
and try to wake up a thread, if necessary, the contention bit is set again
by the operation. This also mitgates the chance that other threads find
the contention bit and try to enter kernel to compete with each other
to wake up sleeping thread, this is unnecessary. With this change, the
mutex owner is no longer holding the mutex until it reaches a point
where kernel umtx queue is locked, it releases the mutex as soon as
possible.
Performance is improved when the mutex is contensted heavily. On Intel
i3-2310M, the runtime of a benchmark program is reduced from 26.87 seconds
to 2.39 seconds, it even is better than UMTX_OP_MUTEX_WAKE which is
deprecated now. http://people.freebsd.org/~davidxu/bench/mutex_perf.c
Stack creation code is changed to call _rtld_get_stack_prot() to get
the stack protection right. There is a race where thread is created
during dlopen() of dso that requires executable stacks. Then,
_rtld_get_stack_prot() may return PROT_READ | PROT_WRITE, but thread
is still not linked into the thread list. In this case, the callback
misses the thread stack, and rechecks the required protection
afterward.
Reviewed by: davidxu
- Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based
condition variable, this should eliminate an extra system call to get
current time.
- Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single
system call. Create userland sleep queue for condition variable, in most
cases, thread will wait in the queue, the pthread_cond_signal will defer
thread wakeup until the mutex is unlocked, it tries to avoid an extra
system call and a extra context switch in time window of pthread_cond_signal
and pthread_mutex_unlock.
The changes are part of process-shared mutex project.
a silly rwlock deadlock problem, the deadlock is caused by writer
waiters, if a thread has already locked a reader lock, and wants to
acquire another reader lock, it will be blocked by writer waiters,
but we had already fixed it years ago.
functions set or get pthread_rwlock type, current supported types are:
PTHREAD_RWLOCK_PREFER_READER_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NP,
default is PTHREAD_RWLOCK_PREFER_WRITER_NONCECURSIVE_NP, this maintains
binary compatible with old code.
