d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6fe65214f2
freebsd-dev/sys/amd64/include/mutex.h

372 lines
10 KiB
C
Raw Normal View History

Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
/*-
* Copyright (c) 1997 Berkeley Software Design, Inc. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. Berkeley Software Design Inc's name may not be used to endorse or
* promote products derived from this software without specific prior
* written permission.
*
* THIS SOFTWARE IS PROVIDED BY BERKELEY SOFTWARE DESIGN INC ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL BERKELEY SOFTWARE DESIGN INC BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* from BSDI $Id: mutex.h,v 2.7.2.35 2000/04/27 03:10:26 cp Exp $
* $FreeBSD$
*/
#ifndef _MACHINE_MUTEX_H_
#define _MACHINE_MUTEX_H_
#ifndef LOCORE
Reduce userland namespace polution.
2000-10-04 01:21:58 +00:00
#ifdef _KERNEL
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
#include <machine/psl.h>
Rename mtx_enter(), mtx_try_enter(), and mtx_exit() and wrap them with cpp macros that expand to pass filename and line number information. This is necessary since we're using inline functions instead of macros now. Add const to the filename pointers passed througout the mtx and witness code.
2000-09-08 21:48:06 +00:00
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
/* Global locks */
- Change fast interrupts on x86 to push a full interrupt frame and to return through doreti to handle ast's. This is necessary for the clock interrupts to work properly. - Change the clock interrupts on the x86 to be fast instead of threaded. This is needed because both hardclock() and statclock() need to run in the context of the current process, not in a separate thread context. - Kill the prevproc hack as it is no longer needed. - We really need Giant when we call psignal(), but we don't want to block during the clock interrupt. Instead, use two p_flag's in the proc struct to mark the current process as having a pending SIGVTALRM or a SIGPROF and let them be delivered during ast() when hardclock() has finished running. - Remove CLKF_BASEPRI, which was #ifdef'd out on the x86 anyways. It was broken on the x86 if it was turned on since cpl is gone. It's only use was to bogusly run softclock() directly during hardclock() rather than scheduling an SWI. - Remove the COM_LOCK simplelock and replace it with a clock_lock spin mutex. Since the spin mutex already handles disabling/restoring interrupts appropriately, this also lets us axe all the *_intr() fu. - Back out the hacks in the APIC_IO x86 cpu_initclocks() code to use temporary fast interrupts for the APIC trial. - Add two new process flags P_ALRMPEND and P_PROFPEND to mark the pending signals in hardclock() that are to be delivered in ast(). Submitted by: jakeb (making statclock safe in a fast interrupt) Submitted by: cp (concept of delaying signals until ast())
2000-10-06 02:20:21 +00:00
extern struct mtx clock_lock;
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
/*
* Debugging
*/
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
#ifdef MUTEX_DEBUG
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
#ifdef _KERN_MUTEX_C_
char STR_IEN[] = "fl & PSL_I";
char STR_IDIS[] = "!(fl & PSL_I)";
char STR_SIEN[] = "mpp->mtx_saveintr & PSL_I";
#else /* _KERN_MUTEX_C_ */
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
extern char STR_IEN[];
extern char STR_IDIS[];
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
extern char STR_SIEN[];
#endif /* _KERN_MUTEX_C_ */
#endif /* MUTEX_DEBUG */
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
#define ASS_IEN MPASS2(read_eflags() & PSL_I, STR_IEN)
#define ASS_IDIS MPASS2((read_eflags() & PSL_I) == 0, STR_IDIS)
#define ASS_SIEN(mpp) MPASS2((mpp)->mtx_saveintr & PSL_I, STR_SIEN)
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
Define the mtx_legal2block() macro used in the witness code that managed to get lost during the MI mutex conversion. Reported by: Steve Kargl <sgk@troutmask.apl.washington.edu>
2000-10-20 22:44:06 +00:00
#define mtx_legal2block() (read_eflags() & PSL_I)
Work around the broken atomic_store_rel_ptr() on the i386 arch by just using atomic_cmpset_rel_ptr() instead for _release_lock_quick(). When atomic_store_rel_ptr() is functional and MP safe, then this can be reverted.
2001-01-14 00:16:17 +00:00
/* Actually release mtx_lock quickly assuming that we own it */
#define _release_lock_quick(mp) \
atomic_cmpset_rel_int(&(mp)->mtx_lock, (mp)->mtx_lock, MTX_UNOWNED)
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
/*
* Assembly macros (for internal use only)
*------------------------------------------------------------------------------
*/
#define _V(x) __STRING(x)
Well, the previous commit wasn't entirely correct either. For now, just disable the optimized mutex micro-operations for the non-I386_CPU case and fall back to the C stubs that call the atomic_foo() inlines.
2000-12-08 05:03:34 +00:00
#if 0
/* #ifndef I386_CPU */
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
/*
* For 486 and newer processors.
*/
/* Get a sleep lock, deal with recursion inline. */
#define _getlock_sleep(mtxp, tid, type) ({ \
int _res; \
\
__asm __volatile ( \
" movl $" _V(MTX_UNOWNED) ",%%eax;" /* Unowned cookie */ \
" " MPLOCKED "" \
" cmpxchgl %3,%1;" /* Try */ \
" jz 1f;" /* Got it */ \
" andl $" _V(MTX_FLAGMASK) ",%%eax;" /* turn off spec bits */ \
" cmpl %%eax,%3;" /* already have it? */ \
" je 2f;" /* yes, recurse */ \
" pushl %4;" \
" pushl %5;" \
" call mtx_enter_hard;" \
" addl $8,%%esp;" \
" jmp 1f;" \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
"2:" \
" " MPLOCKED "" \
" orl $" _V(MTX_RECURSE) ",%1;" \
Rename mtx_enter(), mtx_try_enter(), and mtx_exit() and wrap them with cpp macros that expand to pass filename and line number information. This is necessary since we're using inline functions instead of macros now. Add const to the filename pointers passed througout the mtx and witness code.
2000-09-08 21:48:06 +00:00
" incl %2;" \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
"1:" \
"# getlock_sleep" \
: "=&a" (_res), /* 0 (dummy output) */ \
"+m" (mtxp->mtx_lock), /* 1 */ \
"+m" (mtxp->mtx_recurse) /* 2 */ \
: "r" (tid), /* 3 (input) */ \
"gi" (type), /* 4 */ \
"g" (mtxp) /* 5 */ \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
: "memory", "ecx", "edx" /* used */ ); \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
})
/* Get a spin lock, handle recursion inline (as the less common case) */
#define _getlock_spin_block(mtxp, tid, type) ({ \
int _res; \
\
__asm __volatile ( \
" pushfl;" \
" cli;" \
" movl $" _V(MTX_UNOWNED) ",%%eax;" /* Unowned cookie */ \
" " MPLOCKED "" \
" cmpxchgl %3,%1;" /* Try */ \
" jz 2f;" /* got it */ \
" pushl %4;" \
" pushl %5;" \
" call mtx_enter_hard;" /* mtx_enter_hard(mtxp, type, oflags) */ \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
" addl $12,%%esp;" \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
" jmp 1f;" \
"2: popl %2;" /* save flags */ \
"1:" \
"# getlock_spin_block" \
: "=&a" (_res), /* 0 (dummy output) */ \
"+m" (mtxp->mtx_lock), /* 1 */ \
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
"=m" (mtxp->mtx_saveintr) /* 2 */ \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
: "r" (tid), /* 3 (input) */ \
"gi" (type), /* 4 */ \
"g" (mtxp) /* 5 */ \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
: "memory", "ecx", "edx" /* used */ ); \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
})
/*
* Get a lock without any recursion handling. Calls the hard enter function if
* we can't get it inline.
*/
#define _getlock_norecurse(mtxp, tid, type) ({ \
int _res; \
\
__asm __volatile ( \
" movl $" _V(MTX_UNOWNED) ",%%eax;" /* Unowned cookie */ \
" " MPLOCKED "" \
" cmpxchgl %2,%1;" /* Try */ \
" jz 1f;" /* got it */ \
" pushl %3;" \
" pushl %4;" \
" call mtx_enter_hard;" /* mtx_enter_hard(mtxp, type) */ \
" addl $8,%%esp;" \
"1:" \
"# getlock_norecurse" \
: "=&a" (_res), /* 0 (dummy output) */ \
"+m" (mtxp->mtx_lock) /* 1 */ \
: "r" (tid), /* 2 (input) */ \
"gi" (type), /* 3 */ \
"g" (mtxp) /* 4 */ \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
: "memory", "ecx", "edx" /* used */ ); \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
})
/*
* Release a sleep lock assuming we haven't recursed on it, recursion is handled
* in the hard function.
*/
#define _exitlock_norecurse(mtxp, tid, type) ({ \
int _tid = (int)(tid); \
\
__asm __volatile ( \
" " MPLOCKED "" \
" cmpxchgl %4,%0;" /* try easy rel */ \
" jz 1f;" /* released! */ \
" pushl %2;" \
" pushl %3;" \
" call mtx_exit_hard;" \
" addl $8,%%esp;" \
"1:" \
"# exitlock_norecurse" \
: "+m" (mtxp->mtx_lock), /* 0 */ \
"+a" (_tid) /* 1 */ \
: "gi" (type), /* 2 (input) */ \
"g" (mtxp), /* 3 */ \
"r" (MTX_UNOWNED) /* 4 */ \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
: "memory", "ecx", "edx" /* used */ ); \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
})
/*
* Release a sleep lock when its likely we recursed (the code to
* deal with simple recursion is inline).
*/
#define _exitlock(mtxp, tid, type) ({ \
int _tid = (int)(tid); \
\
__asm __volatile ( \
" " MPLOCKED "" \
" cmpxchgl %5,%0;" /* try easy rel */ \
" jz 1f;" /* released! */ \
" testl $" _V(MTX_RECURSE) ",%%eax;" /* recursed? */ \
" jnz 3f;" /* handle recursion */ \
/* Lock not recursed and contested: do the hard way */ \
" pushl %3;" \
" pushl %4;" \
" call mtx_exit_hard;" /* mtx_exit_hard(mtxp,type) */ \
" addl $8,%%esp;" \
" jmp 1f;" \
/* lock recursed, lower recursion level */ \
Rename mtx_enter(), mtx_try_enter(), and mtx_exit() and wrap them with cpp macros that expand to pass filename and line number information. This is necessary since we're using inline functions instead of macros now. Add const to the filename pointers passed througout the mtx and witness code.
2000-09-08 21:48:06 +00:00
"3: decl %1;" /* one less level */ \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
" jnz 1f;" /* still recursed, done */ \
" lock; andl $~" _V(MTX_RECURSE) ",%0;" /* turn off recurse flag */ \
"1:" \
"# exitlock" \
: "+m" (mtxp->mtx_lock), /* 0 */ \
"+m" (mtxp->mtx_recurse), /* 1 */ \
"+a" (_tid) /* 2 */ \
: "gi" (type), /* 3 (input) */ \
"g" (mtxp), /* 4 */ \
"r" (MTX_UNOWNED) /* 5 */ \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
: "memory", "ecx", "edx" /* used */ ); \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
})
/*
* Release a spin lock (with possible recursion).
*
* We use cmpxchgl to clear lock (instead of simple store) to flush posting
* buffers and make the change visible to other CPU's.
*/
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
#define _exitlock_spin(mtxp) ({ \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
int _res; \
\
__asm __volatile ( \
Fix broken register restraints that needlessly clobbered registers %ecx and %edx resulting in gcc not having enough registers left to work with.
2000-12-07 02:23:16 +00:00
" movl %1,%2;" \
" decl %2;" \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
" js 1f;" \
Fix broken register restraints that needlessly clobbered registers %ecx and %edx resulting in gcc not having enough registers left to work with.
2000-12-07 02:23:16 +00:00
" movl %2,%1;" \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
" jmp 2f;" \
Fix broken register restraints that needlessly clobbered registers %ecx and %edx resulting in gcc not having enough registers left to work with.
2000-12-07 02:23:16 +00:00
"1: movl %0,%2;" \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
" movl $ " _V(MTX_UNOWNED) ",%%ecx;" \
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
" pushl %3;" \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
" " MPLOCKED "" \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
" cmpxchgl %%ecx,%0;" \
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
" popfl;" \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
"2:" \
"# exitlock_spin" \
: "+m" (mtxp->mtx_lock), /* 0 */ \
"+m" (mtxp->mtx_recurse), /* 1 */ \
"=&a" (_res) /* 2 */ \
Argh, revert the clobber changes. Since %ecx and %edx aren't call safe, calling the C functions mtx_enter_hard() and mtx_exit_hard() clobbers them. Note that %eax is also not call safe, but it is already clobbered due to cmpxchg. However, now we are back to not compiling again, so these macros are still left disabled for now.
2000-12-08 18:21:06 +00:00
: "g" (mtxp->mtx_saveintr) /* 3 */ \
: "memory", "ecx" /* used */ ); \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
})
#endif /* I386_CPU */
Rename mtx_enter(), mtx_try_enter(), and mtx_exit() and wrap them with cpp macros that expand to pass filename and line number information. This is necessary since we're using inline functions instead of macros now. Add const to the filename pointers passed througout the mtx and witness code.
2000-09-08 21:48:06 +00:00
#endif /* _KERNEL */
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
#else /* !LOCORE */
/*
Fix a jump to the wrong label, <sigh>. Put a period at the end of a sentence in a comment. Submitted by: bde
2000-12-08 19:53:37 +00:00
* Simple assembly macros to get and release spin locks.
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
*/
Fix the assembly mutex macros to call the appropriate witness functions if the witness code is compiled in. Without this, the witness code doesn't notice that sched_lock is released by fork_trampoline() and thus gets all confused about spin lock order later on.
2000-12-12 03:49:58 +00:00
#ifdef WITNESS
#define WITNESS_ENTER(lck, reg) \
movl lck+MTX_DEBUG,reg; \
cmpl $0,MTXD_WITNESS(reg); \
jz 1f; \
pushl $0; \
pushl $0; \
pushl $MTX_SPIN; \
pushl $lck; \
call witness_enter; \
addl $0x10,%esp; \
1:
#define WITNESS_EXIT(lck, reg) \
movl lck+MTX_DEBUG,reg; \
cmpl $0,MTXD_WITNESS(reg); \
jz 1f; \
pushl $0; \
pushl $0; \
pushl $MTX_SPIN; \
pushl $lck; \
call witness_exit; \
addl $0x10,%esp; \
1:
#else
#define WITNESS_ENTER(lck, reg)
#define WITNESS_EXIT(lck, reg)
#endif
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
#if defined(I386_CPU)
Change the calling conventions of the MTX_ENTER macro to match that of MTX_EXIT. Don't assume that the reg parameter to MTX_ENTER holds curproc, load it explicitly. Put semi-colons at the end of the macros to be more consistent and so its harder to forget them when these change.
2000-12-08 08:49:36 +00:00
#define MTX_ENTER(lck, reg) \
movl _curproc,reg; \
pushfl; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
cli; \
movl reg,lck+MTX_LOCK; \
Fix the assembly mutex macros to call the appropriate witness functions if the witness code is compiled in. Without this, the witness code doesn't notice that sched_lock is released by fork_trampoline() and thus gets all confused about spin lock order later on.
2000-12-12 03:49:58 +00:00
popl lck+MTX_SAVEINTR; \
WITNESS_ENTER(lck, reg)
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
#define MTX_EXIT(lck, reg) \
Fix the assembly mutex macros to call the appropriate witness functions if the witness code is compiled in. Without this, the witness code doesn't notice that sched_lock is released by fork_trampoline() and thus gets all confused about spin lock order later on.
2000-12-12 03:49:58 +00:00
WITNESS_EXIT(lck, reg) \
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
pushl lck+MTX_SAVEINTR; \
Fix the assmebly mutex macros to handle saving/restoring interrupt state properly. Fix the recursive mutex macros to actually compile. At the moment we only use MTX_EXIT anyways.
2000-09-24 23:34:21 +00:00
movl $ MTX_UNOWNED,lck+MTX_LOCK; \
Change the calling conventions of the MTX_ENTER macro to match that of MTX_EXIT. Don't assume that the reg parameter to MTX_ENTER holds curproc, load it explicitly. Put semi-colons at the end of the macros to be more consistent and so its harder to forget them when these change.
2000-12-08 08:49:36 +00:00
popfl;
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
Rename mtx_enter(), mtx_try_enter(), and mtx_exit() and wrap them with cpp macros that expand to pass filename and line number information. This is necessary since we're using inline functions instead of macros now. Add const to the filename pointers passed througout the mtx and witness code.
2000-09-08 21:48:06 +00:00
#else /* I386_CPU */
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
Change the calling conventions of the MTX_ENTER macro to match that of MTX_EXIT. Don't assume that the reg parameter to MTX_ENTER holds curproc, load it explicitly. Put semi-colons at the end of the macros to be more consistent and so its harder to forget them when these change.
2000-12-08 08:49:36 +00:00
#define MTX_ENTER(lck, reg) \
movl _curproc,reg; \
pushfl; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
cli; \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
9: movl $ MTX_UNOWNED,%eax; \
MPLOCKED \
cmpxchgl reg,lck+MTX_LOCK; \
Fix the assmebly mutex macros to handle saving/restoring interrupt state properly. Fix the recursive mutex macros to actually compile. At the moment we only use MTX_EXIT anyways.
2000-09-24 23:34:21 +00:00
jnz 9b; \
Fix the assembly mutex macros to call the appropriate witness functions if the witness code is compiled in. Without this, the witness code doesn't notice that sched_lock is released by fork_trampoline() and thus gets all confused about spin lock order later on.
2000-12-12 03:49:58 +00:00
popl lck+MTX_SAVEINTR; \
WITNESS_ENTER(lck, reg)
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
/* Must use locked bus op (cmpxchg) when setting to unowned (barrier) */
Change the calling conventions of the MTX_ENTER macro to match that of MTX_EXIT. Don't assume that the reg parameter to MTX_ENTER holds curproc, load it explicitly. Put semi-colons at the end of the macros to be more consistent and so its harder to forget them when these change.
2000-12-08 08:49:36 +00:00
#define MTX_EXIT(lck, reg) \
Fix the assembly mutex macros to call the appropriate witness functions if the witness code is compiled in. Without this, the witness code doesn't notice that sched_lock is released by fork_trampoline() and thus gets all confused about spin lock order later on.
2000-12-12 03:49:58 +00:00
WITNESS_EXIT(lck, reg) \
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
pushl lck+MTX_SAVEINTR; \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
movl lck+MTX_LOCK,%eax; \
movl $ MTX_UNOWNED,reg; \
MPLOCKED \
cmpxchgl reg,lck+MTX_LOCK; \
Change the calling conventions of the MTX_ENTER macro to match that of MTX_EXIT. Don't assume that the reg parameter to MTX_ENTER holds curproc, load it explicitly. Put semi-colons at the end of the macros to be more consistent and so its harder to forget them when these change.
2000-12-08 08:49:36 +00:00
popfl;
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
Change the calling conventions of the MTX_ENTER macro to match that of MTX_EXIT. Don't assume that the reg parameter to MTX_ENTER holds curproc, load it explicitly. Put semi-colons at the end of the macros to be more consistent and so its harder to forget them when these change.
2000-12-08 08:49:36 +00:00
#define MTX_ENTER_WITH_RECURSION(lck, reg) \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
pushf; \
cli; \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
movl lck+MTX_LOCK,%eax; \
Fix the assmebly mutex macros to handle saving/restoring interrupt state properly. Fix the recursive mutex macros to actually compile. At the moment we only use MTX_EXIT anyways.
2000-09-24 23:34:21 +00:00
cmpl _curproc,%eax; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
jne 7f; \
incl lck+MTX_RECURSE; \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
jmp 8f; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
7: movl $ MTX_UNOWNED,%eax; \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
MPLOCKED \
cmpxchgl reg,lck+MTX_LOCK; \
Fix a jump to the wrong label, <sigh>. Put a period at the end of a sentence in a comment. Submitted by: bde
2000-12-08 19:53:37 +00:00
jnz 7b; \
- Make the mutex code almost completely machine independent. This greatly reducues the maintenance load for the mutex code. The only MD portions of the mutex code are in machine/mutex.h now, which include the assembly macros for handling mutexes as well as optionally overriding the mutex micro-operations. For example, we use optimized micro-ops on the x86 platform #ifndef I386_CPU. - Change the behavior of the SMP_DEBUG kernel option. In the new code, mtx_assert() only depends on INVARIANTS, allowing other kernel developers to have working mutex assertiions without having to include all of the mutex debugging code. The SMP_DEBUG kernel option has been renamed to MUTEX_DEBUG and now just controls extra mutex debugging code. - Abolish the ugly mtx_f hack. Instead, we dynamically allocate seperate mtx_debug structures on the fly in mtx_init, except for mutexes that are initiated very early in the boot process. These mutexes are declared using a special MUTEX_DECLARE() macro, and use a new flag MTX_COLD when calling mtx_init. This is still somewhat hackish, but it is less evil than the mtx_f filler struct, and the mtx struct is now the same size with and without mutex debugging code. - Add some micro-micro-operation macros for doing the actual atomic operations on the mutex mtx_lock field to make it easier for other archs to override/optimize mutex ops if needed. These new tiny ops also clean up the code in some places by replacing long atomic operation function calls that spanned 2-3 lines with a short 1-line macro call. - Don't call mi_switch() from mtx_enter_hard() when we block while trying to obtain a sleep mutex. Calling mi_switch() would bogusly release Giant before switching to the next process. Instead, inline most of the code from mi_switch() in the mtx_enter_hard() function. Note that when we finally kill Giant we can back this out and go back to calling mi_switch().
2000-10-20 07:26:37 +00:00
popl lck+MTX_SAVEINTR; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
jmp 9f; \
Fix the assmebly mutex macros to handle saving/restoring interrupt state properly. Fix the recursive mutex macros to actually compile. At the moment we only use MTX_EXIT anyways.
2000-09-24 23:34:21 +00:00
8: add $4,%esp; \
Fix the assembly mutex macros to call the appropriate witness functions if the witness code is compiled in. Without this, the witness code doesn't notice that sched_lock is released by fork_trampoline() and thus gets all confused about spin lock order later on.
2000-12-12 03:49:58 +00:00
9: WITNESS_ENTER(lck, reg)
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
Change the calling conventions of the MTX_ENTER macro to match that of MTX_EXIT. Don't assume that the reg parameter to MTX_ENTER holds curproc, load it explicitly. Put semi-colons at the end of the macros to be more consistent and so its harder to forget them when these change.
2000-12-08 08:49:36 +00:00
#define MTX_EXIT_WITH_RECURSION(lck, reg) \
Fix the assembly mutex macros to call the appropriate witness functions if the witness code is compiled in. Without this, the witness code doesn't notice that sched_lock is released by fork_trampoline() and thus gets all confused about spin lock order later on.
2000-12-12 03:49:58 +00:00
WITNESS_EXIT(lck, reg) \
Fix the assmebly mutex macros to handle saving/restoring interrupt state properly. Fix the recursive mutex macros to actually compile. At the moment we only use MTX_EXIT anyways.
2000-09-24 23:34:21 +00:00
movl lck+MTX_RECURSE,%eax; \
Teach MTX_EXIT_RECURSE that the recursion count is a 32-bit integer, not a 16-bit one.
2000-09-22 04:30:33 +00:00
decl %eax; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
js 8f; \
Fix the assmebly mutex macros to handle saving/restoring interrupt state properly. Fix the recursive mutex macros to actually compile. At the moment we only use MTX_EXIT anyways.
2000-09-24 23:34:21 +00:00
movl %eax,lck+MTX_RECURSE; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
jmp 9f; \
8: pushl lck+MTX_SAVEINTR; \
Fix a bug with handling of the saved interrupt state for spin mutexes in the MTX_EXIT_WITH_RECURSION() assembly macro (currently unused). Submitted by: bde
2000-11-13 18:39:18 +00:00
movl lck+MTX_LOCK,%eax; \
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
movl $ MTX_UNOWNED,reg; \
MPLOCKED \
cmpxchgl reg,lck+MTX_LOCK; \
(1) Allow a stray lock prefix to be compiled out with the MPLOCKED macro (2) Use decimal 12 rather than hex 0xc in an addl (3) Implement MTX_ENTER for the I386_CPU case (4) Use semi-colons between instructions to allow MTX_ENTER and MTX_ENTER_WITH_RECURSION to be assembled (5) Use incl instead of incw to increment the recusion count (6) 10 is not a valid label, use 7, 8 and 9 rather than 8, 9 and 10 (7) Sort numeric labels Submitted by: bde (2, 4, and 5)
2000-12-04 12:38:03 +00:00
popf; \
9:
Major update to the way synchronization is done in the kernel. Highlights include: * Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The alpha port is still in transition and currently uses both.) * Per-CPU idle processes. * Interrupts are run in their own separate kernel threads and can be preempted (i386 only). Partially contributed by: BSDi (BSD/OS) Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh
2000-09-07 01:33:02 +00:00
Rename mtx_enter(), mtx_try_enter(), and mtx_exit() and wrap them with cpp macros that expand to pass filename and line number information. This is necessary since we're using inline functions instead of macros now. Add const to the filename pointers passed througout the mtx and witness code.
2000-09-08 21:48:06 +00:00
#endif /* I386_CPU */
#endif /* !LOCORE */
#endif /* __MACHINE_MUTEX_H */
Reference in New Issue Copy Permalink