d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
95a74ab4fb
freebsd-dev/lib/libthr/thread/thr_mutex.c

1201 lines
30 KiB
C
Raw Normal View History

spdx: initial adoption of licensing ID tags. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Special thanks to Wind River for providing access to "The Duke of Highlander" tool: an older (2014) run over FreeBSD tree was useful as a starting point. Initially, only tag files that use BSD 4-Clause "Original" license. RelNotes: yes Differential Revision: https://reviews.freebsd.org/D13133
2017-11-18 14:26:50 +00:00
/*-
* SPDX-License-Identifier: BSD-4-Clause
*
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
* Copyright (c) 1995 John Birrell <jb@cimlogic.com.au>.
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
* Copyright (c) 2006 David Xu <davidxu@freebsd.org>.
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
* Copyright (c) 2015, 2016 The FreeBSD Foundation
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
*
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
* All rights reserved.
*
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
* Portions of this software were developed by Konstantin Belousov
* under sponsorship from the FreeBSD Foundation.
*
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. All advertising materials mentioning features or use of this software
* must display the following acknowledgement:
* This product includes software developed by John Birrell.
* 4. Neither the name of the author nor the names of any co-contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY JOHN BIRRELL AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
Use __FBSDID() for .c files from lib/libthr/thread. Sponsored by: The FreeBSD Foundation
2016-04-08 11:15:26 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
WARNS level 4 cleanup.
2006-04-04 02:57:49 +00:00
#include "namespace.h"
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <sys/param.h>
#include <sys/queue.h>
#include <pthread.h>
Fixed mis-implementation of pthread_mutex_get{spin,yield}loops_np(). Reviewed by: davidxu
2008-03-25 09:48:10 +00:00
#include <pthread_np.h>
WARNS level 4 cleanup.
2006-04-04 02:57:49 +00:00
#include "un-namespace.h"
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
#include "thr_private.h"
Assert that the lock objects put into the off-page, fit into the page. Sponsored by: The FreeBSD Foundation
2016-04-08 10:21:43 +00:00
_Static_assert(sizeof(struct pthread_mutex) <= PAGE_SIZE,
"pthread_mutex is too large for off-page");
Restore revision 1.55, the kris's adaptive mutex type.
2007-10-31 01:37:13 +00:00
/*
* For adaptive mutexes, how many times to spin doing trylock2
* before entering the kernel to block
*/
Increase the default MUTEX_ADAPTIVE_SPINS to 2000, after further testing it turns out 200 was too short to give good adaptive performance. Reviewed by: jeff MFC after: 1 week
2008-04-26 13:19:07 +00:00
#define MUTEX_ADAPTIVE_SPINS 2000
Restore revision 1.55, the kris's adaptive mutex type.
2007-10-31 01:37:13 +00:00
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
/*
* Prototypes
*/
POSIX compliance improvements in the pthread(3) functions. This basically adds makes use of the C99 restrict keyword, and also adds some 'const's to four threading functions: pthread_mutexattr_gettype(), pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018. Hinted by: DragonFlyBSD Relnotes: yes MFC after: 1 month Differential Revision: D16722
2018-08-18 01:05:38 +00:00
int __pthread_mutex_timedlock(pthread_mutex_t * __restrict mutex,
const struct timespec * __restrict abstime);
Add function prototypes.
2007-12-17 02:53:11 +00:00
int _pthread_mutex_getspinloops_np(pthread_mutex_t *mutex, int *count);
int _pthread_mutex_setspinloops_np(pthread_mutex_t *mutex, int count);
int __pthread_mutex_setspinloops_np(pthread_mutex_t *mutex, int count);
int _pthread_mutex_setyieldloops_np(pthread_mutex_t *mutex, int count);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
int _pthread_mutex_getyieldloops_np(pthread_mutex_t *mutex, int *count);
Add function prototypes.
2007-12-17 02:53:11 +00:00
int __pthread_mutex_setyieldloops_np(pthread_mutex_t *mutex, int count);
WARNS level 4 cleanup.
2006-04-04 02:57:49 +00:00
static int mutex_self_trylock(pthread_mutex_t);
static int mutex_self_lock(pthread_mutex_t,
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
const struct timespec *abstime);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
static int mutex_unlock_common(struct pthread_mutex *, bool, int *);
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
static int mutex_lock_sleep(struct pthread *, pthread_mutex_t,
const struct timespec *);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
static void mutex_init_robust(struct pthread *curthread);
static int mutex_qidx(struct pthread_mutex *m);
static bool is_robust_mutex(struct pthread_mutex *m);
static bool is_pshared_mutex(struct pthread_mutex *m);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
__weak_reference(__Tthr_mutex_init, pthread_mutex_init);
__weak_reference(__Tthr_mutex_init, __pthread_mutex_init);
__strong_reference(__Tthr_mutex_init, _pthread_mutex_init);
__weak_reference(__Tthr_mutex_lock, pthread_mutex_lock);
__weak_reference(__Tthr_mutex_lock, __pthread_mutex_lock);
__strong_reference(__Tthr_mutex_lock, _pthread_mutex_lock);
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
__weak_reference(__pthread_mutex_timedlock, pthread_mutex_timedlock);
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
__strong_reference(__pthread_mutex_timedlock, _pthread_mutex_timedlock);
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
__weak_reference(__Tthr_mutex_trylock, pthread_mutex_trylock);
__weak_reference(__Tthr_mutex_trylock, __pthread_mutex_trylock);
__strong_reference(__Tthr_mutex_trylock, _pthread_mutex_trylock);
__weak_reference(_Tthr_mutex_consistent, pthread_mutex_consistent);
__weak_reference(_Tthr_mutex_consistent, _pthread_mutex_consistent);
__strong_reference(_Tthr_mutex_consistent, __pthread_mutex_consistent);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
/* Single underscore versions provided for libc internal usage: */
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
/* No difference between libc and application usage of these: */
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
__weak_reference(_thr_mutex_destroy, pthread_mutex_destroy);
__weak_reference(_thr_mutex_destroy, _pthread_mutex_destroy);
__weak_reference(_thr_mutex_unlock, pthread_mutex_unlock);
__weak_reference(_thr_mutex_unlock, _pthread_mutex_unlock);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
__weak_reference(_pthread_mutex_getprioceiling, pthread_mutex_getprioceiling);
__weak_reference(_pthread_mutex_setprioceiling, pthread_mutex_setprioceiling);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
__weak_reference(__pthread_mutex_setspinloops_np, pthread_mutex_setspinloops_np);
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
__strong_reference(__pthread_mutex_setspinloops_np, _pthread_mutex_setspinloops_np);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
__weak_reference(_pthread_mutex_getspinloops_np, pthread_mutex_getspinloops_np);
__weak_reference(__pthread_mutex_setyieldloops_np, pthread_mutex_setyieldloops_np);
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
__strong_reference(__pthread_mutex_setyieldloops_np, _pthread_mutex_setyieldloops_np);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
__weak_reference(_pthread_mutex_getyieldloops_np, pthread_mutex_getyieldloops_np);
Per discussion on -threads, rename _islocked_np() to _isowned_np().
2008-02-06 19:34:31 +00:00
__weak_reference(_pthread_mutex_isowned_np, pthread_mutex_isowned_np);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
static void
mutex_init_link(struct pthread_mutex *m)
{
#if defined(_PTHREADS_INVARIANTS)
m->m_qe.tqe_prev = NULL;
m->m_qe.tqe_next = NULL;
m->m_pqe.tqe_prev = NULL;
m->m_pqe.tqe_next = NULL;
#endif
}
static void
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
mutex_assert_is_owned(struct pthread_mutex *m __unused)
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
{
#if defined(_PTHREADS_INVARIANTS)
libthr: Use formatted PANIC() No functional change, although _thread_printf() may be slightly less functional or render some values differently from libc snprintf(3). No ABI change. Reviewed by: kib Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6672
2016-06-01 16:12:26 +00:00
if (__predict_false(m->m_qe.tqe_prev == NULL))
PANIC("mutex %p own %#x is not on list %p %p",
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
m, m->m_lock.m_owner, m->m_qe.tqe_prev, m->m_qe.tqe_next);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
#endif
}
static void
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
mutex_assert_not_owned(struct pthread *curthread __unused,
struct pthread_mutex *m __unused)
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
{
#if defined(_PTHREADS_INVARIANTS)
if (__predict_false(m->m_qe.tqe_prev != NULL ||
libthr: Use formatted PANIC() No functional change, although _thread_printf() may be slightly less functional or render some values differently from libc snprintf(3). No ABI change. Reviewed by: kib Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6672
2016-06-01 16:12:26 +00:00
m->m_qe.tqe_next != NULL))
PANIC("mutex %p own %#x is on list %p %p",
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
m, m->m_lock.m_owner, m->m_qe.tqe_prev, m->m_qe.tqe_next);
if (__predict_false(is_robust_mutex(m) &&
(m->m_lock.m_rb_lnk != 0 || m->m_rb_prev != NULL ||
(is_pshared_mutex(m) && curthread->robust_list ==
(uintptr_t)&m->m_lock) ||
(!is_pshared_mutex(m) && curthread->priv_robust_list ==
libthr: Use formatted PANIC() No functional change, although _thread_printf() may be slightly less functional or render some values differently from libc snprintf(3). No ABI change. Reviewed by: kib Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6672
2016-06-01 16:12:26 +00:00
(uintptr_t)&m->m_lock))))
PANIC(
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
"mutex %p own %#x is on robust linkage %p %p head %p phead %p",
m, m->m_lock.m_owner, (void *)m->m_lock.m_rb_lnk,
m->m_rb_prev, (void *)curthread->robust_list,
(void *)curthread->priv_robust_list);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
#endif
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
static bool
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
is_pshared_mutex(struct pthread_mutex *m)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
return ((m->m_lock.m_flags & USYNC_PROCESS_SHARED) != 0);
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
static bool
is_robust_mutex(struct pthread_mutex *m)
{
return ((m->m_lock.m_flags & UMUTEX_ROBUST) != 0);
}
int
_mutex_enter_robust(struct pthread *curthread, struct pthread_mutex *m)
{
#if defined(_PTHREADS_INVARIANTS)
if (__predict_false(curthread->inact_mtx != 0))
PANIC("inact_mtx enter");
#endif
if (!is_robust_mutex(m))
return (0);
mutex_init_robust(curthread);
curthread->inact_mtx = (uintptr_t)&m->m_lock;
return (1);
}
void
_mutex_leave_robust(struct pthread *curthread, struct pthread_mutex *m __unused)
{
#if defined(_PTHREADS_INVARIANTS)
if (__predict_false(curthread->inact_mtx != (uintptr_t)&m->m_lock))
PANIC("inact_mtx leave");
#endif
curthread->inact_mtx = 0;
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
static int
mutex_check_attr(const struct pthread_mutex_attr *attr)
{
if (attr->m_type < PTHREAD_MUTEX_ERRORCHECK ||
attr->m_type >= PTHREAD_MUTEX_TYPE_MAX)
return (EINVAL);
if (attr->m_protocol < PTHREAD_PRIO_NONE ||
attr->m_protocol > PTHREAD_PRIO_PROTECT)
return (EINVAL);
return (0);
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
static void
mutex_init_robust(struct pthread *curthread)
{
struct umtx_robust_lists_params rb;
if (curthread == NULL)
curthread = _get_curthread();
if (curthread->robust_inited)
return;
rb.robust_list_offset = (uintptr_t)&curthread->robust_list;
rb.robust_priv_list_offset = (uintptr_t)&curthread->priv_robust_list;
rb.robust_inact_offset = (uintptr_t)&curthread->inact_mtx;
_umtx_op(NULL, UMTX_OP_ROBUST_LISTS, sizeof(rb), &rb, NULL);
curthread->robust_inited = 1;
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
static void
mutex_init_body(struct pthread_mutex *pmutex,
const struct pthread_mutex_attr *attr)
{
Reimplement mutex_init to get rid of compile warning.
2006-02-28 06:06:19 +00:00
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
pmutex->m_flags = attr->m_type;
Reimplement mutex_init to get rid of compile warning.
2006-02-28 06:06:19 +00:00
pmutex->m_count = 0;
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
pmutex->m_spinloops = 0;
pmutex->m_yieldloops = 0;
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
mutex_init_link(pmutex);
switch (attr->m_protocol) {
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
case PTHREAD_PRIO_NONE:
pmutex->m_lock.m_owner = UMUTEX_UNOWNED;
pmutex->m_lock.m_flags = 0;
break;
Use umutex APIs to implement pthread_mutex, member pp_mutexq is added into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex, no real mutex code is changed, the mutex locking and unlocking code should has same performance as before.
2006-08-28 04:52:50 +00:00
case PTHREAD_PRIO_INHERIT:
pmutex->m_lock.m_owner = UMUTEX_UNOWNED;
pmutex->m_lock.m_flags = UMUTEX_PRIO_INHERIT;
break;
case PTHREAD_PRIO_PROTECT:
pmutex->m_lock.m_owner = UMUTEX_CONTESTED;
pmutex->m_lock.m_flags = UMUTEX_PRIO_PROTECT;
pmutex->m_lock.m_ceilings[0] = attr->m_ceiling;
break;
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (attr->m_pshared == PTHREAD_PROCESS_SHARED)
pmutex->m_lock.m_flags |= USYNC_PROCESS_SHARED;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (attr->m_robust == PTHREAD_MUTEX_ROBUST) {
mutex_init_robust(NULL);
pmutex->m_lock.m_flags |= UMUTEX_ROBUST;
}
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
if (PMUTEX_TYPE(pmutex->m_flags) == PTHREAD_MUTEX_ADAPTIVE_NP) {
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
pmutex->m_spinloops =
_thr_spinloops ? _thr_spinloops: MUTEX_ADAPTIVE_SPINS;
pmutex->m_yieldloops = _thr_yieldloops;
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
}
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
static int
mutex_init(pthread_mutex_t *mutex,
const struct pthread_mutex_attr *mutex_attr,
void *(calloc_cb)(size_t, size_t))
{
const struct pthread_mutex_attr *attr;
struct pthread_mutex *pmutex;
int error;
if (mutex_attr == NULL) {
attr = &_pthread_mutexattr_default;
} else {
attr = mutex_attr;
error = mutex_check_attr(attr);
if (error != 0)
return (error);
}
if ((pmutex = (pthread_mutex_t)
calloc_cb(1, sizeof(struct pthread_mutex))) == NULL)
return (ENOMEM);
mutex_init_body(pmutex, attr);
Reimplement mutex_init to get rid of compile warning.
2006-02-28 06:06:19 +00:00
*mutex = pmutex;
return (0);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
static int
init_static(struct pthread *thread, pthread_mutex_t *mutex)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
int ret;
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
THR_LOCK_ACQUIRE(thread, &_mutex_static_lock);
Refactor _pthread_mutex_init o Simplify the logic by removing a lot of unnecesary nesting o Reduce the amount of local variables o Zero-out the allocated structure and get rid of all the unnecessary setting to 0 and NULL; Refactor _pthread_mutex_destroy o Simplify the logic by removing a lot of unnecesary nesting o No need to check pointer that the mutex attributes points to. Checking passed in pointer is enough.
2004-01-19 15:00:57 +00:00
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
if (*mutex == THR_MUTEX_INITIALIZER)
Untangle jemalloc and mutexes initialization. The need to use libc malloc(3) from some places in libthr always caused issues. For instance, per-thread key allocation was switched to use plain mmap(2) to get storage, because some third party mallocs used keys for implementation of calloc(3). Even more important, libthr calls calloc(3) during initialization of pthread mutexes, and jemalloc uses pthread mutexes. Jemalloc provides some way to both postpone the initialization, and to make initialization to use specialized allocator, but this is very fragile and often breaks. See the referenced PR for another example. Add the small malloc implementation used by rtld, to libthr. Use it in thr_spec.c and for mutexes initialization. This avoids the issues with mutual dependencies between malloc and libthr in principle. The drawback is that some more allocations are not interceptable for alternate malloc implementations. There should be not too much memory use from this allocator, and the alternative, direct use of mmap(2) is obviously worse. PR: 235211 MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D18988
2019-01-29 22:46:44 +00:00
ret = mutex_init(mutex, &_pthread_mutexattr_default,
__thr_calloc);
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
else if (*mutex == THR_ADAPTIVE_MUTEX_INITIALIZER)
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = mutex_init(mutex, &_pthread_mutexattr_adaptive_default,
Untangle jemalloc and mutexes initialization. The need to use libc malloc(3) from some places in libthr always caused issues. For instance, per-thread key allocation was switched to use plain mmap(2) to get storage, because some third party mallocs used keys for implementation of calloc(3). Even more important, libthr calls calloc(3) during initialization of pthread mutexes, and jemalloc uses pthread mutexes. Jemalloc provides some way to both postpone the initialization, and to make initialization to use specialized allocator, but this is very fragile and often breaks. See the referenced PR for another example. Add the small malloc implementation used by rtld, to libthr. Use it in thr_spec.c and for mutexes initialization. This avoids the issues with mutual dependencies between malloc and libthr in principle. The drawback is that some more allocations are not interceptable for alternate malloc implementations. There should be not too much memory use from this allocator, and the alternative, direct use of mmap(2) is obviously worse. PR: 235211 MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D18988
2019-01-29 22:46:44 +00:00
__thr_calloc);
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
else
ret = 0;
THR_LOCK_RELEASE(thread, &_mutex_static_lock);
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
return (ret);
}
Move code calculating new inherited priority into single function.
2006-11-11 13:33:47 +00:00
static void
set_inherited_priority(struct pthread *curthread, struct pthread_mutex *m)
{
struct pthread_mutex *m2;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
m2 = TAILQ_LAST(&curthread->mq[mutex_qidx(m)], mutex_queue);
Move code calculating new inherited priority into single function.
2006-11-11 13:33:47 +00:00
if (m2 != NULL)
m->m_lock.m_ceilings[1] = m2->m_lock.m_ceilings[0];
else
m->m_lock.m_ceilings[1] = -1;
}
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
static void
shared_mutex_init(struct pthread_mutex *pmtx, const struct
pthread_mutex_attr *mutex_attr)
{
static const struct pthread_mutex_attr foobar_mutex_attr = {
.m_type = PTHREAD_MUTEX_DEFAULT,
.m_protocol = PTHREAD_PRIO_NONE,
.m_ceiling = 0,
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
.m_pshared = PTHREAD_PROCESS_SHARED,
.m_robust = PTHREAD_MUTEX_STALLED,
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
};
bool done;
/*
* Hack to allow multiple pthread_mutex_init() calls on the
* same process-shared mutex. We rely on kernel allocating
* zeroed offpage for the mutex, i.e. the
* PMUTEX_INITSTAGE_ALLOC value must be zero.
*/
for (done = false; !done;) {
switch (pmtx->m_ps) {
case PMUTEX_INITSTAGE_DONE:
atomic_thread_fence_acq();
done = true;
break;
case PMUTEX_INITSTAGE_ALLOC:
if (atomic_cmpset_int(&pmtx->m_ps,
PMUTEX_INITSTAGE_ALLOC, PMUTEX_INITSTAGE_BUSY)) {
if (mutex_attr == NULL)
mutex_attr = &foobar_mutex_attr;
mutex_init_body(pmtx, mutex_attr);
atomic_store_rel_int(&pmtx->m_ps,
PMUTEX_INITSTAGE_DONE);
done = true;
}
break;
case PMUTEX_INITSTAGE_BUSY:
_pthread_yield();
break;
default:
PANIC("corrupted offpage");
break;
}
}
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
int
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
__Tthr_mutex_init(pthread_mutex_t * __restrict mutex,
POSIX compliance improvements in the pthread(3) functions. This basically adds makes use of the C99 restrict keyword, and also adds some 'const's to four threading functions: pthread_mutexattr_gettype(), pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018. Hinted by: DragonFlyBSD Relnotes: yes MFC after: 1 month Differential Revision: D16722
2018-08-18 01:05:38 +00:00
const pthread_mutexattr_t * __restrict mutex_attr)
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread_mutex *pmtx;
int ret;
Ensure that threading library is initialized in pthread_mutex_init(). We need at least thr_malloc ready. The situation is possible e.g. in case of libthr being listed in DT_NEEDED before some of its consumers. Reported and tested by: lev Sponsored by: The FreeBSD Foundation MFC after: 1 week
2020-11-30 17:00:36 +00:00
_thr_check_init();
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (mutex_attr != NULL) {
ret = mutex_check_attr(*mutex_attr);
if (ret != 0)
return (ret);
}
if (mutex_attr == NULL ||
(*mutex_attr)->m_pshared == PTHREAD_PROCESS_PRIVATE) {
Fixes for very early use of the pthread_mutex_* and libthr malloc. When libthr is statically linked into the binary, order of the constructors execution is not deterministic. It is possible for the application constructor to use pthread_mutex_* functions before the libthr initialization was done. Handle it by: - making thr_malloc.c locking functions operational when curthread is not yet set; - making __thr_malloc_init() idempotent, allowing more than one call to it; - unconditionally calling __thr_malloc_init() before initializing a process-private mutex. Reported and tested by: mmel Sponsored by: The FreeBSD Foundation MFC after: 1 week
2019-02-04 21:16:15 +00:00
__thr_malloc_init();
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
return (mutex_init(mutex, mutex_attr ? *mutex_attr : NULL,
Untangle jemalloc and mutexes initialization. The need to use libc malloc(3) from some places in libthr always caused issues. For instance, per-thread key allocation was switched to use plain mmap(2) to get storage, because some third party mallocs used keys for implementation of calloc(3). Even more important, libthr calls calloc(3) during initialization of pthread mutexes, and jemalloc uses pthread mutexes. Jemalloc provides some way to both postpone the initialization, and to make initialization to use specialized allocator, but this is very fragile and often breaks. See the referenced PR for another example. Add the small malloc implementation used by rtld, to libthr. Use it in thr_spec.c and for mutexes initialization. This avoids the issues with mutual dependencies between malloc and libthr in principle. The drawback is that some more allocations are not interceptable for alternate malloc implementations. There should be not too much memory use from this allocator, and the alternative, direct use of mmap(2) is obviously worse. PR: 235211 MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D18988
2019-01-29 22:46:44 +00:00
__thr_calloc));
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
}
POSIX compliance improvements in the pthread(3) functions. This basically adds makes use of the C99 restrict keyword, and also adds some 'const's to four threading functions: pthread_mutexattr_gettype(), pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018. Hinted by: DragonFlyBSD Relnotes: yes MFC after: 1 month Differential Revision: D16722
2018-08-18 01:05:38 +00:00
pmtx = __thr_pshared_offpage(__DECONST(void *, mutex), 1);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (pmtx == NULL)
return (EFAULT);
*mutex = THR_PSHARED_PTR;
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
shared_mutex_init(pmtx, *mutex_attr);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
return (0);
Add _pthread_mutex_init_calloc_cb() to libthr and libkse, so that malloc(3) (part of libc) can use pthreads mutexes without causing infinite recursion during initialization.
2007-11-27 03:16:44 +00:00
}
/* This function is used internally by malloc. */
int
_pthread_mutex_init_calloc_cb(pthread_mutex_t *mutex,
void *(calloc_cb)(size_t, size_t))
{
static const struct pthread_mutex_attr attr = {
Fix pointer dereferencing problems in _pthread_mutex_init_calloc_cb() that were obscured by pseudo-opaque pthreads API pointer casting.
2007-11-28 00:16:24 +00:00
.m_type = PTHREAD_MUTEX_NORMAL,
Add _pthread_mutex_init_calloc_cb() to libthr and libkse, so that malloc(3) (part of libc) can use pthreads mutexes without causing infinite recursion during initialization.
2007-11-27 03:16:44 +00:00
.m_protocol = PTHREAD_PRIO_NONE,
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
.m_ceiling = 0,
.m_pshared = PTHREAD_PROCESS_PRIVATE,
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
.m_robust = PTHREAD_MUTEX_STALLED,
Add _pthread_mutex_init_calloc_cb() to libthr and libkse, so that malloc(3) (part of libc) can use pthreads mutexes without causing infinite recursion during initialization.
2007-11-27 03:16:44 +00:00
};
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
int ret;
Add _pthread_mutex_init_calloc_cb() to libthr and libkse, so that malloc(3) (part of libc) can use pthreads mutexes without causing infinite recursion during initialization.
2007-11-27 03:16:44 +00:00
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
ret = mutex_init(mutex, &attr, calloc_cb);
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
if (ret == 0)
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
(*mutex)->m_flags |= PMUTEX_FLAG_PRIVATE;
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
return (ret);
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
}
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
/*
* Fix mutex ownership for child process.
*
* Process private mutex ownership is transmitted from the forking
* thread to the child process.
*
* Process shared mutex should not be inherited because owner is
* forking thread which is in parent process, they are removed from
* the owned mutex list.
*/
static void
queue_fork(struct pthread *curthread, struct mutex_queue *q,
struct mutex_queue *qp, uint bit)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
struct pthread_mutex *m;
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
TAILQ_INIT(q);
TAILQ_FOREACH(m, qp, m_pqe) {
TAILQ_INSERT_TAIL(q, m, m_qe);
m->m_lock.m_owner = TID(curthread) | bit;
}
}
void
_mutex_fork(struct pthread *curthread)
{
Use umutex APIs to implement pthread_mutex, member pp_mutexq is added into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex, no real mutex code is changed, the mutex locking and unlocking code should has same performance as before.
2006-08-28 04:52:50 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
queue_fork(curthread, &curthread->mq[TMQ_NORM],
&curthread->mq[TMQ_NORM_PRIV], 0);
queue_fork(curthread, &curthread->mq[TMQ_NORM_PP],
&curthread->mq[TMQ_NORM_PP_PRIV], UMUTEX_CONTESTED);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
queue_fork(curthread, &curthread->mq[TMQ_ROBUST_PP],
&curthread->mq[TMQ_ROBUST_PP_PRIV], UMUTEX_CONTESTED);
curthread->robust_list = 0;
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
int
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
_thr_mutex_destroy(pthread_mutex_t *mutex)
o Refactor and, among other things, get rid of insane nesting levels. o Fix mutex priority protocols. Keep separate counts of priority inheritance and protection mutexes to make things easier. This will not have much affect since this is only the userland side, and the rest involves kernel scheduling.
2004-02-18 15:16:31 +00:00
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
pthread_mutex_t m, m1;
Remove locking and unlock in pthread_mutex_destroy, because it can not fix race condition in application code, as a result, the problem described in PR threads/151767 is avoided.
2010-10-27 04:19:07 +00:00
int ret;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
m = *mutex;
if (m < THR_MUTEX_DESTROYED) {
ret = 0;
} else if (m == THR_MUTEX_DESTROYED) {
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
ret = EINVAL;
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
} else {
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (m == THR_PSHARED_PTR) {
m1 = __thr_pshared_offpage(mutex, 0);
if (m1 != NULL) {
Fix destruction of the robust mutexes. If robust mutex' owner terminated, causing kernel-assisted state recovery, and then pthread_mutex_destroy() is executed as the next action, assert is triggered about mutex still being on the list. Ignore the mutex linkage in pthread_mutex_destroy() for shared robust mutexes with dead owner, same as for enqueue_mutex(). Reported by: avg Sponsored by: The FreeBSD Foundation MFC after: 1 week
2019-09-23 13:24:31 +00:00
if ((uint32_t)m1->m_lock.m_owner !=
UMUTEX_RB_OWNERDEAD) {
mutex_assert_not_owned(
_get_curthread(), m1);
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
__thr_pshared_destroy(mutex);
}
*mutex = THR_MUTEX_DESTROYED;
return (0);
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (PMUTEX_OWNER_ID(m) != 0 &&
(uint32_t)m->m_lock.m_owner != UMUTEX_RB_NOTRECOV) {
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
ret = EBUSY;
} else {
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
*mutex = THR_MUTEX_DESTROYED;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
mutex_assert_not_owned(_get_curthread(), m);
Untangle jemalloc and mutexes initialization. The need to use libc malloc(3) from some places in libthr always caused issues. For instance, per-thread key allocation was switched to use plain mmap(2) to get storage, because some third party mallocs used keys for implementation of calloc(3). Even more important, libthr calls calloc(3) during initialization of pthread mutexes, and jemalloc uses pthread mutexes. Jemalloc provides some way to both postpone the initialization, and to make initialization to use specialized allocator, but this is very fragile and often breaks. See the referenced PR for another example. Add the small malloc implementation used by rtld, to libthr. Use it in thr_spec.c and for mutexes initialization. This avoids the issues with mutual dependencies between malloc and libthr in principle. The drawback is that some more allocations are not interceptable for alternate malloc implementations. There should be not too much memory use from this allocator, and the alternative, direct use of mmap(2) is obviously worse. PR: 235211 MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D18988
2019-01-29 22:46:44 +00:00
__thr_free(m);
Remove locking and unlock in pthread_mutex_destroy, because it can not fix race condition in application code, as a result, the problem described in PR threads/151767 is avoided.
2010-10-27 04:19:07 +00:00
ret = 0;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
}
}
return (ret);
o Refactor and, among other things, get rid of insane nesting levels. o Fix mutex priority protocols. Keep separate counts of priority inheritance and protection mutexes to make things easier. This will not have much affect since this is only the userland side, and the rest involves kernel scheduling.
2004-02-18 15:16:31 +00:00
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
static int
mutex_qidx(struct pthread_mutex *m)
{
if ((m->m_lock.m_flags & UMUTEX_PRIO_PROTECT) == 0)
return (TMQ_NORM);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
return (is_robust_mutex(m) ? TMQ_ROBUST_PP : TMQ_NORM_PP);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
/*
* Both enqueue_mutex() and dequeue_mutex() operate on the
* thread-private linkage of the locked mutexes and on the robust
* linkage.
*
* Robust list, as seen by kernel, must be consistent even in the case
* of thread termination at arbitrary moment. Since either enqueue or
* dequeue for list walked by kernel consists of rewriting a single
* forward pointer, it is safe. On the other hand, rewrite of the
* back pointer is not atomic WRT the forward one, but kernel does not
* care.
*/
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
static void
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
enqueue_mutex(struct pthread *curthread, struct pthread_mutex *m,
int error)
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
struct pthread_mutex *m1;
uintptr_t *rl;
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
int qidx;
/* Add to the list of owned mutexes: */
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (error != EOWNERDEAD)
mutex_assert_not_owned(curthread, m);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
qidx = mutex_qidx(m);
TAILQ_INSERT_TAIL(&curthread->mq[qidx], m, m_qe);
if (!is_pshared_mutex(m))
TAILQ_INSERT_TAIL(&curthread->mq[qidx + 1], m, m_pqe);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (is_robust_mutex(m)) {
rl = is_pshared_mutex(m) ? &curthread->robust_list :
&curthread->priv_robust_list;
m->m_rb_prev = NULL;
if (*rl != 0) {
m1 = __containerof((void *)*rl,
struct pthread_mutex, m_lock);
m->m_lock.m_rb_lnk = (uintptr_t)&m1->m_lock;
m1->m_rb_prev = m;
} else {
m1 = NULL;
m->m_lock.m_rb_lnk = 0;
}
*rl = (uintptr_t)&m->m_lock;
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
}
static void
dequeue_mutex(struct pthread *curthread, struct pthread_mutex *m)
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
struct pthread_mutex *mp, *mn;
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
int qidx;
mutex_assert_is_owned(m);
qidx = mutex_qidx(m);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (is_robust_mutex(m)) {
mp = m->m_rb_prev;
if (mp == NULL) {
if (is_pshared_mutex(m)) {
curthread->robust_list = m->m_lock.m_rb_lnk;
} else {
curthread->priv_robust_list =
m->m_lock.m_rb_lnk;
}
} else {
mp->m_lock.m_rb_lnk = m->m_lock.m_rb_lnk;
}
if (m->m_lock.m_rb_lnk != 0) {
mn = __containerof((void *)m->m_lock.m_rb_lnk,
struct pthread_mutex, m_lock);
mn->m_rb_prev = m->m_rb_prev;
}
m->m_lock.m_rb_lnk = 0;
m->m_rb_prev = NULL;
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
TAILQ_REMOVE(&curthread->mq[qidx], m, m_qe);
if (!is_pshared_mutex(m))
TAILQ_REMOVE(&curthread->mq[qidx + 1], m, m_pqe);
if ((m->m_lock.m_flags & UMUTEX_PRIO_PROTECT) != 0)
set_inherited_priority(curthread, m);
mutex_init_link(m);
}
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
static int
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
check_and_init_mutex(pthread_mutex_t *mutex, struct pthread_mutex **m)
o Refactor and, among other things, get rid of insane nesting levels. o Fix mutex priority protocols. Keep separate counts of priority inheritance and protection mutexes to make things easier. This will not have much affect since this is only the userland side, and the rest involves kernel scheduling.
2004-02-18 15:16:31 +00:00
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
int ret;
*m = *mutex;
ret = 0;
if (*m == THR_PSHARED_PTR) {
*m = __thr_pshared_offpage(mutex, 0);
if (*m == NULL)
ret = EINVAL;
If off-page lookup failed, there is no memory to perform shared_mutex_init() upon. Sponsored by: The FreeBSD Foundation
2016-04-12 10:25:44 +00:00
else
shared_mutex_init(*m, NULL);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
} else if (__predict_false(*m <= THR_MUTEX_DESTROYED)) {
if (*m == THR_MUTEX_DESTROYED) {
ret = EINVAL;
} else {
ret = init_static(_get_curthread(), mutex);
if (ret == 0)
*m = *mutex;
}
}
return (ret);
}
int
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
__Tthr_mutex_trylock(pthread_mutex_t *mutex)
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
{
struct pthread *curthread;
struct pthread_mutex *m;
Use umutex APIs to implement pthread_mutex, member pp_mutexq is added into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex, no real mutex code is changed, the mutex locking and unlocking code should has same performance as before.
2006-08-28 04:52:50 +00:00
uint32_t id;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
int ret, robust;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = check_and_init_mutex(mutex, &m);
if (ret != 0)
return (ret);
curthread = _get_curthread();
Use umutex APIs to implement pthread_mutex, member pp_mutexq is added into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex, no real mutex code is changed, the mutex locking and unlocking code should has same performance as before.
2006-08-28 04:52:50 +00:00
id = TID(curthread);
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
if (m->m_flags & PMUTEX_FLAG_PRIVATE)
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
THR_CRITICAL_ENTER(curthread);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
robust = _mutex_enter_robust(curthread, m);
Use umutex APIs to implement pthread_mutex, member pp_mutexq is added into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex, no real mutex code is changed, the mutex locking and unlocking code should has same performance as before.
2006-08-28 04:52:50 +00:00
ret = _thr_umutex_trylock(&m->m_lock, id);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (__predict_true(ret == 0) || ret == EOWNERDEAD) {
enqueue_mutex(curthread, m, ret);
if (ret == EOWNERDEAD)
m->m_lock.m_flags |= UMUTEX_NONCONSISTENT;
} else if (PMUTEX_OWNER_ID(m) == id) {
WARNS level 4 cleanup.
2006-04-04 02:57:49 +00:00
ret = mutex_self_trylock(m);
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
} /* else {} */
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (robust)
_mutex_leave_robust(curthread, m);
Fix __pthread_mutex_trylock() to call THR_CRITICAL_LEAVE() on failure rather than on success. This regression was introduced by r300043 (Add implementation of robust mutexes...). MFC after: 1 day
2017-05-13 17:49:53 +00:00
if (ret != 0 && ret != EOWNERDEAD &&
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
(m->m_flags & PMUTEX_FLAG_PRIVATE) != 0)
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
THR_CRITICAL_LEAVE(curthread);
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
return (ret);
o Refactor and, among other things, get rid of insane nesting levels. o Fix mutex priority protocols. Keep separate counts of priority inheritance and protection mutexes to make things easier. This will not have much affect since this is only the userland side, and the rest involves kernel scheduling.
2004-02-18 15:16:31 +00:00
}
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
static int
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
mutex_lock_sleep(struct pthread *curthread, struct pthread_mutex *m,
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
const struct timespec *abstime)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
uint32_t id, owner;
int count, ret;
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
id = TID(curthread);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (PMUTEX_OWNER_ID(m) == id)
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
return (mutex_self_lock(m, abstime));
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
/*
* For adaptive mutexes, spin for a bit in the expectation
* that if the application requests this mutex type then
* the lock is likely to be released quickly and it is
* faster than entering the kernel
*/
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (__predict_false((m->m_lock.m_flags & (UMUTEX_PRIO_PROTECT |
UMUTEX_PRIO_INHERIT | UMUTEX_ROBUST | UMUTEX_NONCONSISTENT)) != 0))
goto sleep_in_kernel;
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
if (!_thr_is_smp)
goto yield_loop;
count = m->m_spinloops;
while (count--) {
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
owner = m->m_lock.m_owner;
if ((owner & ~UMUTEX_CONTESTED) == 0) {
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (atomic_cmpset_acq_32(&m->m_lock.m_owner, owner,
id | owner)) {
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
ret = 0;
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
goto done;
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
}
Add a new "non-portable" mutex type, PTHREAD_MUTEX_ADAPTIVE_NP. This is also implemented in glibc and is used by a number of existing applications (mysql, firefox, etc). This mutex type is a default mutex with the additional property that it spins briefly when attempting to acquire a contested lock, doing trylock operations in userland before entering the kernel to block if eventually unsuccessful. The expectation is that applications requesting this mutex type know that the mutex is likely to be only held for very brief periods, so it is faster to spin in userland and probably succeed in acquiring the mutex, than to enter the kernel and sleep, only to be woken up almost immediately. This can help significantly in certain cases when pthread mutexes are heavily contended and held for brief durations (such as mysql). Spin up to 200 times before entering the kernel, which represents only a few us on modern CPUs. No performance degradation was observed with this value and it is sufficient to avoid a large performance drop in mysql performance in the heavily contended pthread mutex case. The libkse implementation is a NOP. Reviewed by: jeff MFC after: 3 days
2007-10-29 21:01:47 +00:00
}
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
CPU_SPINWAIT;
}
Add a new "non-portable" mutex type, PTHREAD_MUTEX_ADAPTIVE_NP. This is also implemented in glibc and is used by a number of existing applications (mysql, firefox, etc). This mutex type is a default mutex with the additional property that it spins briefly when attempting to acquire a contested lock, doing trylock operations in userland before entering the kernel to block if eventually unsuccessful. The expectation is that applications requesting this mutex type know that the mutex is likely to be only held for very brief periods, so it is faster to spin in userland and probably succeed in acquiring the mutex, than to enter the kernel and sleep, only to be woken up almost immediately. This can help significantly in certain cases when pthread mutexes are heavily contended and held for brief durations (such as mysql). Spin up to 200 times before entering the kernel, which represents only a few us on modern CPUs. No performance degradation was observed with this value and it is sufficient to avoid a large performance drop in mysql performance in the heavily contended pthread mutex case. The libkse implementation is a NOP. Reviewed by: jeff MFC after: 3 days
2007-10-29 21:01:47 +00:00
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
yield_loop:
count = m->m_yieldloops;
while (count--) {
_sched_yield();
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
owner = m->m_lock.m_owner;
if ((owner & ~UMUTEX_CONTESTED) == 0) {
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (atomic_cmpset_acq_32(&m->m_lock.m_owner, owner,
id | owner)) {
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
ret = 0;
goto done;
}
}
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
}
sleep_in_kernel:
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (abstime == NULL)
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
ret = __thr_umutex_lock(&m->m_lock, id);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
else if (__predict_false(abstime->tv_nsec < 0 ||
abstime->tv_nsec >= 1000000000))
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
ret = EINVAL;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
else
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
ret = __thr_umutex_timedlock(&m->m_lock, id, abstime);
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
done:
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (ret == 0 || ret == EOWNERDEAD) {
enqueue_mutex(curthread, m, ret);
if (ret == EOWNERDEAD)
m->m_lock.m_flags |= UMUTEX_NONCONSISTENT;
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
return (ret);
o Refactor and, among other things, get rid of insane nesting levels. o Fix mutex priority protocols. Keep separate counts of priority inheritance and protection mutexes to make things easier. This will not have much affect since this is only the userland side, and the rest involves kernel scheduling.
2004-02-18 15:16:31 +00:00
}
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
static inline int
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
mutex_lock_common(struct pthread_mutex *m, const struct timespec *abstime,
bool cvattach, bool rb_onlist)
o Refactor and, among other things, get rid of insane nesting levels. o Fix mutex priority protocols. Keep separate counts of priority inheritance and protection mutexes to make things easier. This will not have much affect since this is only the userland side, and the rest involves kernel scheduling.
2004-02-18 15:16:31 +00:00
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
struct pthread *curthread;
int ret, robust;
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
libthr: fix warnings from GCC when WARNS=6 Fix warnings about: - redundant declarations - a local variable shadowing a global function (dlinfo) - an old-style function definition (with an empty parameter list) - a variable that is possibly used uninitialized "make tinderbox" passes this time, except for a few unrelated kernel failures. Reviewed by: kib MFC after: 3 days Sponsored by: Dell EMC Differential Revision: https://reviews.freebsd.org/D10870
2017-05-23 16:12:50 +00:00
robust = 0; /* pacify gcc */
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
curthread = _get_curthread();
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
if (!cvattach && m->m_flags & PMUTEX_FLAG_PRIVATE)
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
THR_CRITICAL_ENTER(curthread);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (!rb_onlist)
robust = _mutex_enter_robust(curthread, m);
ret = _thr_umutex_trylock2(&m->m_lock, TID(curthread));
if (ret == 0 || ret == EOWNERDEAD) {
enqueue_mutex(curthread, m, ret);
if (ret == EOWNERDEAD)
m->m_lock.m_flags |= UMUTEX_NONCONSISTENT;
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
} else {
ret = mutex_lock_sleep(curthread, m, abstime);
Add two commands to _umtx_op system call to allow a simple mutex to be locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff
2008-06-24 07:32:12 +00:00
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (!rb_onlist && robust)
_mutex_leave_robust(curthread, m);
if (ret != 0 && ret != EOWNERDEAD &&
(m->m_flags & PMUTEX_FLAG_PRIVATE) != 0 && !cvattach)
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
THR_CRITICAL_LEAVE(curthread);
return (ret);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
}
int
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
__Tthr_mutex_lock(pthread_mutex_t *mutex)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread_mutex *m;
int ret;
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
_thr_check_init();
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = check_and_init_mutex(mutex, &m);
if (ret == 0)
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
ret = mutex_lock_common(m, NULL, false, false);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
return (ret);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
}
int
POSIX compliance improvements in the pthread(3) functions. This basically adds makes use of the C99 restrict keyword, and also adds some 'const's to four threading functions: pthread_mutexattr_gettype(), pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018. Hinted by: DragonFlyBSD Relnotes: yes MFC after: 1 month Differential Revision: D16722
2018-08-18 01:05:38 +00:00
__pthread_mutex_timedlock(pthread_mutex_t * __restrict mutex,
const struct timespec * __restrict abstime)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread_mutex *m;
int ret;
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
_thr_check_init();
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = check_and_init_mutex(mutex, &m);
if (ret == 0)
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
ret = mutex_lock_common(m, abstime, false, false);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
return (ret);
o Implement pthread_mutex_timedlock(), which does not block indefinitely on a mutex locked by another thread. o document it: pthread_mutex_timedlock(3)
2003-12-30 08:44:55 +00:00
}
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
int
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
_thr_mutex_unlock(pthread_mutex_t *mutex)
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
{
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
struct pthread_mutex *mp;
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (*mutex == THR_PSHARED_PTR) {
mp = __thr_pshared_offpage(mutex, 0);
if (mp == NULL)
return (EINVAL);
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
shared_mutex_init(mp, NULL);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
} else {
mp = *mutex;
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
return (mutex_unlock_common(mp, false, NULL));
Begin making libthr async signal safe. Create a private, single underscore, version of pthread_mutex_unlock for libc. pthread_mutex_lock already has one. These versions are different from the ones that applications will link against because they block all signals from the time a call to lock the mutex is made until it is successfully unlocked.
2003-07-02 02:05:23 +00:00
}
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
int
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
_mutex_cv_lock(struct pthread_mutex *m, int count, bool rb_onlist)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
int error;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
error = mutex_lock_common(m, NULL, true, rb_onlist);
if (error == 0 || error == EOWNERDEAD)
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
m->m_count = count;
return (error);
}
int
MFp4: Further decreases unexpected context switches by defering mutex wakeup until internal sleep queue lock is released.
2012-08-11 23:17:02 +00:00
_mutex_cv_unlock(struct pthread_mutex *m, int *count, int *defer)
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
{
/*
* Clear the count in case this is a recursive mutex.
*/
*count = m->m_count;
m->m_count = 0;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
(void)mutex_unlock_common(m, true, defer);
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
return (0);
}
int
_mutex_cv_attach(struct pthread_mutex *m, int count)
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
struct pthread *curthread;
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
curthread = _get_curthread();
enqueue_mutex(curthread, m, 0);
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
m->m_count = count;
Return 0 instead of garbage value. Found by: clang static analyzer
2011-01-06 08:13:30 +00:00
return (0);
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
}
int
_mutex_cv_detach(struct pthread_mutex *mp, int *recurse)
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
struct pthread *curthread;
int deferred, error;
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
curthread = _get_curthread();
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
if ((error = _mutex_owned(curthread, mp)) != 0)
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
return (error);
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
/*
* Clear the count in case this is a recursive mutex.
*/
*recurse = mp->m_count;
mp->m_count = 0;
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
dequeue_mutex(curthread, mp);
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
/* Will this happen in real-world ? */
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if ((mp->m_flags & PMUTEX_FLAG_DEFERRED) != 0) {
deferred = 1;
mp->m_flags &= ~PMUTEX_FLAG_DEFERRED;
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
} else
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
deferred = 0;
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (deferred) {
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
_thr_wake_all(curthread->defer_waiters,
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
curthread->nwaiter_defer);
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
curthread->nwaiter_defer = 0;
Do not check validity of timeout if a mutex can be acquired immediately. Completly drop recursive mutex in pthread_cond_wait and restore recursive after resumption. Reorganize code to make gcc to generate better code.
2006-04-08 13:24:44 +00:00
}
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
return (0);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
static int
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
mutex_self_trylock(struct pthread_mutex *m)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
int ret;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
switch (PMUTEX_TYPE(m->m_flags)) {
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
case PTHREAD_MUTEX_ERRORCHECK:
For pthread_mutex_trylock() call on owned error-check or non-portable adaptive mutex, return EDEADLK as required by POSIX. The pthread_mutex_lock() is already compliant. Tested by: Guy Yur <guyyur@gmail.com> Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Approved by: re (gjb)
2016-06-25 11:30:40 +00:00
case PTHREAD_MUTEX_NORMAL:
Revert r302194, there are issues with some applications after changing the return value, in particular console-kit-daemon. Reported by: Ivan Klymenko <fidaj@ukr.net> Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Approved by: re (gjb)
2016-06-25 20:20:24 +00:00
case PTHREAD_MUTEX_ADAPTIVE_NP:
For pthread_mutex_trylock() call on owned error-check or non-portable adaptive mutex, return EDEADLK as required by POSIX. The pthread_mutex_lock() is already compliant. Tested by: Guy Yur <guyyur@gmail.com> Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Approved by: re (gjb)
2016-06-25 11:30:40 +00:00
ret = EBUSY;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
break;
case PTHREAD_MUTEX_RECURSIVE:
/* Increment the lock count: */
if (m->m_count + 1 > 0) {
m->m_count++;
ret = 0;
} else
ret = EAGAIN;
break;
default:
/* Trap invalid mutex types; */
ret = EINVAL;
}
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
return (ret);
}
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
static int
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
mutex_self_lock(struct pthread_mutex *m, const struct timespec *abstime)
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
{
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
struct timespec ts1, ts2;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
int ret;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
switch (PMUTEX_TYPE(m->m_flags)) {
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
case PTHREAD_MUTEX_ERRORCHECK:
Adaptive mutexes should have the same deadlock detection properties that default (errorcheck) mutexes do. Noticed by: davidxu
2007-10-30 09:24:23 +00:00
case PTHREAD_MUTEX_ADAPTIVE_NP:
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
if (abstime) {
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
if (abstime->tv_sec < 0 || abstime->tv_nsec < 0 ||
abstime->tv_nsec >= 1000000000) {
ret = EINVAL;
} else {
clock_gettime(CLOCK_REALTIME, &ts1);
TIMESPEC_SUB(&ts2, abstime, &ts1);
__sys_nanosleep(&ts2, NULL);
ret = ETIMEDOUT;
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
} else {
/*
* POSIX specifies that mutexes should return
* EDEADLK if a recursive lock is detected.
*/
ret = EDEADLK;
}
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
break;
case PTHREAD_MUTEX_NORMAL:
/*
* What SS2 define as a 'normal' mutex. Intentionally
* deadlock on attempts to get a lock you already own.
*/
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
ret = 0;
if (abstime) {
- Reduce function call overhead for uncontended case. - Remove unused flags MUTEX_FLAGS_* and their code. - Check validity of the timeout parameter in mutex_self_lock().
2008-05-29 07:57:33 +00:00
if (abstime->tv_sec < 0 || abstime->tv_nsec < 0 ||
abstime->tv_nsec >= 1000000000) {
ret = EINVAL;
} else {
clock_gettime(CLOCK_REALTIME, &ts1);
TIMESPEC_SUB(&ts2, abstime, &ts1);
__sys_nanosleep(&ts2, NULL);
ret = ETIMEDOUT;
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
} else {
ts1.tv_sec = 30;
ts1.tv_nsec = 0;
for (;;)
__sys_nanosleep(&ts1, NULL);
}
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
break;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
case PTHREAD_MUTEX_RECURSIVE:
/* Increment the lock count: */
if (m->m_count + 1 > 0) {
m->m_count++;
ret = 0;
} else
ret = EAGAIN;
q§
2004-05-20 11:55:04 +00:00
break;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
default:
/* Trap invalid mutex types; */
ret = EINVAL;
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
return (ret);
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
}
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
static int
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
mutex_unlock_common(struct pthread_mutex *m, bool cv, int *mtx_defer)
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
struct pthread *curthread;
Use umutex APIs to implement pthread_mutex, member pp_mutexq is added into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex, no real mutex code is changed, the mutex locking and unlocking code should has same performance as before.
2006-08-28 04:52:50 +00:00
uint32_t id;
Do not access mutex memory after unlock. PR: 237195 Reported by: freebsd@hurrikhan.eu Sponsored by: The FreeBSD Foundation MFC after: 1 week
2019-04-12 17:27:19 +00:00
int deferred, error, private, robust;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
if (__predict_false(m <= THR_MUTEX_DESTROYED)) {
if (m == THR_MUTEX_DESTROYED)
return (EINVAL);
return (EPERM);
}
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
curthread = _get_curthread();
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
id = TID(curthread);
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
/*
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
* Check if the running thread is not the owner of the mutex.
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
*/
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (__predict_false(PMUTEX_OWNER_ID(m) != id))
Do not check validity of timeout if a mutex can be acquired immediately. Completly drop recursive mutex in pthread_cond_wait and restore recursive after resumption. Reorganize code to make gcc to generate better code.
2006-04-08 13:24:44 +00:00
return (EPERM);
Propagate errors from _thr_umutex_unlock2 through mutex_unlock_common. Errors from _thr_umutex_unlock2 should "never happen" in normal circumstances. If they do, however, return them to the application so it can fail early and loudly. Hiding the errors will only delay the inevitable failure, making it harder to find and diagnose. Submitted by: Eric van Gyzen <eric_van_gyzen@dell.com> Obtained from: Dell Inc. PR: 198914 MFC after: 1 week
2015-02-25 16:18:26 +00:00
error = 0;
Do not access mutex memory after unlock. PR: 237195 Reported by: freebsd@hurrikhan.eu Sponsored by: The FreeBSD Foundation MFC after: 1 week
2019-04-12 17:27:19 +00:00
private = (m->m_flags & PMUTEX_FLAG_PRIVATE) != 0;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (__predict_false(PMUTEX_TYPE(m->m_flags) ==
PTHREAD_MUTEX_RECURSIVE && m->m_count > 0)) {
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
m->m_count--;
Import my recent 1:1 threading working. some features improved includes: 1. fast simple type mutex. 2. __thread tls works. 3. asynchronous cancellation works ( using signal ). 4. thread synchronization is fully based on umtx, mainly, condition variable and other synchronization objects were rewritten by using umtx directly. those objects can be shared between processes via shared memory, it has to change ABI which does not happen yet. 5. default stack size is increased to 1M on 32 bits platform, 2M for 64 bits platform. As the result, some mysql super-smack benchmarks show performance is improved massivly. Okayed by: jeff, mtm, rwatson, scottl
2005-04-02 01:20:00 +00:00
} else {
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if ((m->m_flags & PMUTEX_FLAG_DEFERRED) != 0) {
deferred = 1;
m->m_flags &= ~PMUTEX_FLAG_DEFERRED;
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
} else
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
deferred = 0;
Check invalid mutex in _mutex_cv_unlock.
2010-09-29 06:06:58 +00:00
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
robust = _mutex_enter_robust(curthread, m);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
dequeue_mutex(curthread, m);
Propagate errors from _thr_umutex_unlock2 through mutex_unlock_common. Errors from _thr_umutex_unlock2 should "never happen" in normal circumstances. If they do, however, return them to the application so it can fail early and loudly. Hiding the errors will only delay the inevitable failure, making it harder to find and diagnose. Submitted by: Eric van Gyzen <eric_van_gyzen@dell.com> Obtained from: Dell Inc. PR: 198914 MFC after: 1 week
2015-02-25 16:18:26 +00:00
error = _thr_umutex_unlock2(&m->m_lock, id, mtx_defer);
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (deferred) {
if (mtx_defer == NULL) {
_thr_wake_all(curthread->defer_waiters,
curthread->nwaiter_defer);
curthread->nwaiter_defer = 0;
} else
*mtx_defer = 1;
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (robust)
_mutex_leave_robust(curthread, m);
Use umutex APIs to implement pthread_mutex, member pp_mutexq is added into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex, no real mutex code is changed, the mutex locking and unlocking code should has same performance as before.
2006-08-28 04:52:50 +00:00
}
Do not access mutex memory after unlock. PR: 237195 Reported by: freebsd@hurrikhan.eu Sponsored by: The FreeBSD Foundation MFC after: 1 week
2019-04-12 17:27:19 +00:00
if (!cv && private)
Change atfork lock from mutex to rwlock, also make mutexes used by malloc() module private type, when private type mutex is locked/unlocked, thread critical region is entered or leaved. These changes makes fork() async-signal safe which required by POSIX. Note that user's atfork handler still needs to be async-signal safe, but it is not problem of libthr, it is user's responsiblity.
2010-09-01 03:11:21 +00:00
THR_CRITICAL_LEAVE(curthread);
Propagate errors from _thr_umutex_unlock2 through mutex_unlock_common. Errors from _thr_umutex_unlock2 should "never happen" in normal circumstances. If they do, however, return them to the application so it can fail early and loudly. Hiding the errors will only delay the inevitable failure, making it harder to find and diagnose. Submitted by: Eric van Gyzen <eric_van_gyzen@dell.com> Obtained from: Dell Inc. PR: 198914 MFC after: 1 week
2015-02-25 16:18:26 +00:00
return (error);
- Add libthr but don't hook it up to the regular build yet. This is an adaptation of libc_r for the thr system call interface. This is beta quality code.
2003-04-01 03:46:29 +00:00
}
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
int
POSIX compliance improvements in the pthread(3) functions. This basically adds makes use of the C99 restrict keyword, and also adds some 'const's to four threading functions: pthread_mutexattr_gettype(), pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018. Hinted by: DragonFlyBSD Relnotes: yes MFC after: 1 month Differential Revision: D16722
2018-08-18 01:05:38 +00:00
_pthread_mutex_getprioceiling(const pthread_mutex_t * __restrict mutex,
int * __restrict prioceiling)
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
{
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
struct pthread_mutex *m;
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (*mutex == THR_PSHARED_PTR) {
POSIX compliance improvements in the pthread(3) functions. This basically adds makes use of the C99 restrict keyword, and also adds some 'const's to four threading functions: pthread_mutexattr_gettype(), pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018. Hinted by: DragonFlyBSD Relnotes: yes MFC after: 1 month Differential Revision: D16722
2018-08-18 01:05:38 +00:00
m = __thr_pshared_offpage(__DECONST(void *, mutex), 0);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (m == NULL)
return (EINVAL);
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
shared_mutex_init(m, NULL);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
} else {
m = *mutex;
if (m <= THR_MUTEX_DESTROYED)
return (EINVAL);
WARNS level 4 cleanup.
2006-04-04 02:57:49 +00:00
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if ((m->m_lock.m_flags & UMUTEX_PRIO_PROTECT) == 0)
return (EINVAL);
*prioceiling = m->m_lock.m_ceilings[0];
return (0);
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
}
Remove priority mutex code because it does not work correctly, to make it work, turnstile like mechanism to support priority propagating and other realtime scheduling options in kernel should be available to userland mutex, for the moment, I just want to make libthr be simple and efficient thread library. Discussed with: deischen, julian
2006-03-27 23:50:21 +00:00
int
POSIX compliance improvements in the pthread(3) functions. This basically adds makes use of the C99 restrict keyword, and also adds some 'const's to four threading functions: pthread_mutexattr_gettype(), pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018. Hinted by: DragonFlyBSD Relnotes: yes MFC after: 1 month Differential Revision: D16722
2018-08-18 01:05:38 +00:00
_pthread_mutex_setprioceiling(pthread_mutex_t * __restrict mutex,
int ceiling, int * __restrict old_ceiling)
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread *curthread;
Check if the PP mutex is recursive, if we have already locked it, place the mutex in right order sorted by priority ceiling.
2007-01-05 03:29:15 +00:00
struct pthread_mutex *m, *m1, *m2;
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct mutex_queue *q, *qp;
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
int qidx, ret;
- Restore old mutex code from libc_r. It is more standards compliant. This was changed because originally we were blocking on the umtx and allowing the kernel to do the queueing. It was decided that the lib should queue and start the threads in the order it decides and the umtx code would just be used like spinlocks.
2003-04-01 22:39:31 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (*mutex == THR_PSHARED_PTR) {
m = __thr_pshared_offpage(mutex, 0);
if (m == NULL)
return (EINVAL);
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
shared_mutex_init(m, NULL);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
} else {
m = *mutex;
if (m <= THR_MUTEX_DESTROYED)
return (EINVAL);
}
if ((m->m_lock.m_flags & UMUTEX_PRIO_PROTECT) == 0)
Check if the PP mutex is recursive, if we have already locked it, place the mutex in right order sorted by priority ceiling.
2007-01-05 03:29:15 +00:00
return (EINVAL);
ret = __thr_umutex_set_ceiling(&m->m_lock, ceiling, old_ceiling);
if (ret != 0)
return (ret);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
curthread = _get_curthread();
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (PMUTEX_OWNER_ID(m) == TID(curthread)) {
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
mutex_assert_is_owned(m);
Check if the PP mutex is recursive, if we have already locked it, place the mutex in right order sorted by priority ceiling.
2007-01-05 03:29:15 +00:00
m1 = TAILQ_PREV(m, mutex_queue, m_qe);
m2 = TAILQ_NEXT(m, m_qe);
Convert ceiling type to unsigned integer before comparing, fix compiler warnings.
2007-11-21 05:25:27 +00:00
if ((m1 != NULL && m1->m_lock.m_ceilings[0] > (u_int)ceiling) ||
(m2 != NULL && m2->m_lock.m_ceilings[0] < (u_int)ceiling)) {
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
qidx = mutex_qidx(m);
q = &curthread->mq[qidx];
qp = &curthread->mq[qidx + 1];
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
TAILQ_REMOVE(q, m, m_qe);
if (!is_pshared_mutex(m))
TAILQ_REMOVE(qp, m, m_pqe);
TAILQ_FOREACH(m2, q, m_qe) {
Convert ceiling type to unsigned integer before comparing, fix compiler warnings.
2007-11-21 05:25:27 +00:00
if (m2->m_lock.m_ceilings[0] > (u_int)ceiling) {
Check if the PP mutex is recursive, if we have already locked it, place the mutex in right order sorted by priority ceiling.
2007-01-05 03:29:15 +00:00
TAILQ_INSERT_BEFORE(m2, m, m_qe);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (!is_pshared_mutex(m)) {
while (m2 != NULL &&
is_pshared_mutex(m2)) {
m2 = TAILQ_PREV(m2,
mutex_queue, m_qe);
}
if (m2 == NULL) {
TAILQ_INSERT_HEAD(qp,
m, m_pqe);
} else {
TAILQ_INSERT_BEFORE(m2,
m, m_pqe);
}
}
Check if the PP mutex is recursive, if we have already locked it, place the mutex in right order sorted by priority ceiling.
2007-01-05 03:29:15 +00:00
return (0);
}
}
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
TAILQ_INSERT_TAIL(q, m, m_qe);
if (!is_pshared_mutex(m))
TAILQ_INSERT_TAIL(qp, m, m_pqe);
Check if the PP mutex is recursive, if we have already locked it, place the mutex in right order sorted by priority ceiling.
2007-01-05 03:29:15 +00:00
}
}
return (0);
msg1
2003-05-12 10:34:01 +00:00
}
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
int
_pthread_mutex_getspinloops_np(pthread_mutex_t *mutex, int *count)
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread_mutex *m;
int ret;
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = check_and_init_mutex(mutex, &m);
if (ret == 0)
*count = m->m_spinloops;
return (ret);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
}
int
__pthread_mutex_setspinloops_np(pthread_mutex_t *mutex, int count)
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread_mutex *m;
int ret;
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = check_and_init_mutex(mutex, &m);
if (ret == 0)
m->m_spinloops = count;
return (ret);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
}
int
_pthread_mutex_getyieldloops_np(pthread_mutex_t *mutex, int *count)
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread_mutex *m;
int ret;
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = check_and_init_mutex(mutex, &m);
if (ret == 0)
*count = m->m_yieldloops;
return (ret);
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
}
int
__pthread_mutex_setyieldloops_np(pthread_mutex_t *mutex, int count)
{
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
struct pthread_mutex *m;
int ret;
In current code, statically initialized and destroyed object have same null value, the code can not distinguish between them, to fix the problem, now a destroyed object is assigned to a non-null value, and it will be rejected by some pthread functions. PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that adaptive mutex can be statically initialized correctly.
2010-09-28 04:57:56 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
ret = check_and_init_mutex(mutex, &m);
if (ret == 0)
m->m_yieldloops = count;
1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin loop count. 2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield loop count. 3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.
2007-12-14 06:25:57 +00:00
return (0);
}
Add pthread_mutex_islocked_np(), a cheap way to verify that a mutex is locked. This is intended primarily to support the userland equivalent of the various *_ASSERT_LOCKED() macros we have in the kernel. MFC after: 2 weeks
2008-02-03 22:38:10 +00:00
int
Per discussion on -threads, rename _islocked_np() to _isowned_np().
2008-02-06 19:34:31 +00:00
_pthread_mutex_isowned_np(pthread_mutex_t *mutex)
Add pthread_mutex_islocked_np(), a cheap way to verify that a mutex is locked. This is intended primarily to support the userland equivalent of the various *_ASSERT_LOCKED() macros we have in the kernel. MFC after: 2 weeks
2008-02-03 22:38:10 +00:00
{
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
struct pthread_mutex *m;
Add pthread_mutex_islocked_np(), a cheap way to verify that a mutex is locked. This is intended primarily to support the userland equivalent of the various *_ASSERT_LOCKED() macros we have in the kernel. MFC after: 2 weeks
2008-02-03 22:38:10 +00:00
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
if (*mutex == THR_PSHARED_PTR) {
m = __thr_pshared_offpage(mutex, 0);
if (m == NULL)
return (0);
Apparently there are some popular programs around which assume that it is safe to call pthread_mutex_init() on the same shared mutex several times. POSIX claims that the behaviour in this case is undefined. Make this working by only allowing one caller to initialize the mutex. Other callers either see already completed initialization and do nothing, or busy-loop yielding while designated initializer finishes. Also make the API requirements loose by initializing mutexes on other pthread_mutex*() calls if they see uninitialized shared mutex. Only mutexes provide the hack for now, but it could be also implemented for other process shared primitives from libthr. Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com> Sponsored by: The FreeBSD Foundation
2016-03-22 10:51:42 +00:00
shared_mutex_init(m, NULL);
Implement process-shared locks support for libthr.so.3, without breaking the ABI. Special value is stored in the lock pointer to indicate shared lock, and offline page in the shared memory is allocated to store the actual lock. Reviewed by: vangyzen (previous version) Discussed with: deischen, emaste, jhb, rwatson, Martin Simmons <martin@lispworks.com> Tested by: pho Sponsored by: The FreeBSD Foundation
2016-02-28 17:52:33 +00:00
} else {
m = *mutex;
if (m <= THR_MUTEX_DESTROYED)
return (0);
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
return (PMUTEX_OWNER_ID(m) == TID(_get_curthread()));
Add pthread_mutex_islocked_np(), a cheap way to verify that a mutex is locked. This is intended primarily to support the userland equivalent of the various *_ASSERT_LOCKED() macros we have in the kernel. MFC after: 2 weeks
2008-02-03 22:38:10 +00:00
}
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
int
_mutex_owned(struct pthread *curthread, const struct pthread_mutex *mp)
{
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
if (__predict_false(mp <= THR_MUTEX_DESTROYED)) {
if (mp == THR_MUTEX_DESTROYED)
return (EINVAL);
return (EPERM);
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
if (PMUTEX_OWNER_ID(mp) != TID(curthread))
MFp4: - Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based condition variable, this should eliminate an extra system call to get current time. - Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single system call. Create userland sleep queue for condition variable, in most cases, thread will wait in the queue, the pthread_cond_signal will defer thread wakeup until the mutex is unlocked, it tries to avoid an extra system call and a extra context switch in time window of pthread_cond_signal and pthread_mutex_unlock. The changes are part of process-shared mutex project.
2010-12-22 05:01:52 +00:00
return (EPERM);
return (0);
}
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
int
Avoid conflicts with libc symbols in libthr jump table. In some corner cases of static linking and unexpected libraries order on the linker command line, libc symbol might preempt the same libthr symbol, in which case libthr jump table points back to libc causing either infinite recursion or loop. Handle all of such symbols by using private libthr names for them, ensuring that the right pointers are installed into the table. In collaboration with: arichardson PR: 239475 Tested by: pho MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21088
2019-07-31 19:27:20 +00:00
_Tthr_mutex_consistent(pthread_mutex_t *mutex)
Add implementation of robust mutexes, hopefully close enough to the intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013. A robust mutex is guaranteed to be cleared by the system upon either thread or process owner termination while the mutex is held. The next mutex locker is then notified about inconsistent mutex state and can execute (or abandon) corrective actions. The patch mostly consists of small changes here and there, adding neccessary checks for the inconsistent and abandoned conditions into existing paths. Additionally, the thread exit handler was extended to iterate over the userspace-maintained list of owned robust mutexes, unlocking and marking as terminated each of them. The list of owned robust mutexes cannot be maintained atomically synchronous with the mutex lock state (it is possible in kernel, but is too expensive). Instead, for the duration of lock or unlock operation, the current mutex is remembered in a special slot that is also checked by the kernel at thread termination. Kernel must be aware about the per-thread location of the heads of robust mutex lists and the current active mutex slot. When a thread touches a robust mutex for the first time, a new umtx op syscall is issued which informs about location of lists heads. The umtx sleep queues for PP and PI mutexes are split between non-robust and robust. Somewhat unrelated changes in the patch: 1. Style. 2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared pi mutexes. 3. Removal of the userspace struct pthread_mutex m_owner field. 4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls the lifetime of the shared mutex associated with a vnode' page. Reviewed by: jilles (previous version, supposedly the objection was fixed) Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects) Tested by: pho Sponsored by: The FreeBSD Foundation
2016-05-17 09:56:22 +00:00
{
struct pthread_mutex *m;
struct pthread *curthread;
if (*mutex == THR_PSHARED_PTR) {
m = __thr_pshared_offpage(mutex, 0);
if (m == NULL)
return (EINVAL);
shared_mutex_init(m, NULL);
} else {
m = *mutex;
if (m <= THR_MUTEX_DESTROYED)
return (EINVAL);
}
curthread = _get_curthread();
if ((m->m_lock.m_flags & (UMUTEX_ROBUST | UMUTEX_NONCONSISTENT)) !=
(UMUTEX_ROBUST | UMUTEX_NONCONSISTENT))
return (EINVAL);
if (PMUTEX_OWNER_ID(m) != TID(curthread))
return (EPERM);
m->m_lock.m_flags &= ~UMUTEX_NONCONSISTENT;
return (0);
}
Reference in New Issue Copy Permalink