d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
a9463ba804
freebsd-dev/sys/kern/kern_lock.c

639 lines
17 KiB
C
Raw Normal View History

Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Copyright (c) 1995
* The Regents of the University of California. All rights reserved.
*
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
* Copyright (C) 1997
* John S. Dyson. All rights reserved.
*
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
* This code contains ideas from software contributed to Berkeley by
* Avadis Tevanian, Jr., Michael Wayne Young, and the Mach Operating
* System project at Carnegie-Mellon University.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. All advertising materials mentioning features or use of this software
* must display the following acknowledgement:
* This product includes software developed by the University of
* California, Berkeley and its contributors.
* 4. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)kern_lock.c 8.18 (Berkeley) 5/21/95
$Id$ -> $FreeBSD$
1999-08-28 01:08:13 +00:00
* $FreeBSD$
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
*/
#include <sys/param.h>
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
#include <sys/kernel.h>
Add missing includes of sys/ktr.h.
2001-10-11 17:53:43 +00:00
#include <sys/ktr.h>
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
#include <sys/lock.h>
Include <sys/lockmgr.h> for the definitions of the locking interfaces that are implemented here instead of depending on namespace pollution in <sys/lock.h>. Fixed nearby include messes (1 disordered include and 1 unused include).
2002-08-27 09:59:47 +00:00
#include <sys/lockmgr.h>
- machine/mutex.h -> sys/mutex.h - The initial lock_mtx mutex used in the lockmgr code is initialized very early, so use MUTEX_DECLARE() and MTX_COLD.
2000-10-20 07:28:00 +00:00
#include <sys/mutex.h>
Include <sys/lockmgr.h> for the definitions of the locking interfaces that are implemented here instead of depending on namespace pollution in <sys/lock.h>. Fixed nearby include messes (1 disordered include and 1 unused include).
2002-08-27 09:59:47 +00:00
#include <sys/proc.h>
Replace original rev 1.2; Author: mpp; Date: 1997/02/12 06:52:30 Add missing #include <sys/systm.h>
1997-03-25 16:36:35 +00:00
#include <sys/systm.h>
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Locking primitives implementation.
* Locks provide shared/exclusive sychronization.
*/
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
#define LOCK_WAIT_TIME 100
#define LOCK_SAMPLE_WAIT 7
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
#if defined(DIAGNOSTIC)
#define LOCK_INLINE
#else
Really finish supporting compiling with `gcc -ansi'.
1998-04-17 04:53:44 +00:00
#define LOCK_INLINE __inline
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
#define LK_ALL (LK_HAVE_EXCL | LK_WANT_EXCL | LK_WANT_UPGRADE | \
LK_SHARE_NONZERO | LK_WAIT_NONZERO)
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
/*
* Mutex array variables. Rather than each lockmgr lock having its own mutex,
* share a fixed (at boot time) number of mutexes across all lockmgr locks in
* order to keep sizeof(struct lock) down.
*/
Create a mutex pool API for short term leaf mutexes. Replace the manual mutex pool in kern_lock.c (lockmgr locks) with the new API. Replace the mutexes embedded in sxlocks with the new API.
2001-11-13 21:55:13 +00:00
int lock_mtx_valid;
Remove MUTEX_DECLARE() and MTX_COLD. Instead, postpone full mutex initialization until after malloc() is safe to call, then iterate through all mutexes and complete their initialization. This change is necessary in order to avoid some circular bootstrapping dependencies.
2001-01-21 07:52:20 +00:00
static struct mtx lock_mtx;
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
static int acquire(struct lock **lkpp, int extflags, int wanted);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
static int apause(struct lock *lkp, int flags);
static int acquiredrain(struct lock *lkp, int extflags) ;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
static void
lockmgr_init(void *dummy __unused)
{
/*
* Initialize the lockmgr protection mutex if it hasn't already been
* done. Unless something changes about kernel startup order, VM
* initialization will always cause this mutex to already be
* initialized in a call to lockinit().
*/
Create a mutex pool API for short term leaf mutexes. Replace the manual mutex pool in kern_lock.c (lockmgr locks) with the new API. Replace the mutexes embedded in sxlocks with the new API.
2001-11-13 21:55:13 +00:00
if (lock_mtx_valid == 0) {
Change callers of mtx_init() to pass in an appropriate lock type name. In most cases NULL is passed, but in some cases such as network driver locks (which use the MTX_NETWORK_LOCK macro) and UMA zone locks, a name is used. Tested on: i386, alpha, sparc64
2002-04-04 21:03:38 +00:00
mtx_init(&lock_mtx, "lockmgr", NULL, MTX_DEF);
Create a mutex pool API for short term leaf mutexes. Replace the manual mutex pool in kern_lock.c (lockmgr locks) with the new API. Replace the mutexes embedded in sxlocks with the new API.
2001-11-13 21:55:13 +00:00
lock_mtx_valid = 1;
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
}
}
SYSINIT(lmgrinit, SI_SUB_LOCK, SI_ORDER_FIRST, lockmgr_init, NULL)
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
static LOCK_INLINE void
sharelock(struct lock *lkp, int incr) {
lkp->lk_flags |= LK_SHARE_NONZERO;
lkp->lk_sharecount += incr;
}
static LOCK_INLINE void
shareunlock(struct lock *lkp, int decr) {
KNFize, by bde.
1999-01-10 01:58:29 +00:00
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
KASSERT(lkp->lk_sharecount >= decr, ("shareunlock: count < decr"));
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
if (lkp->lk_sharecount == decr) {
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
lkp->lk_flags &= ~LK_SHARE_NONZERO;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
if (lkp->lk_flags & (LK_WANT_UPGRADE | LK_WANT_EXCL)) {
wakeup(lkp);
}
lkp->lk_sharecount = 0;
} else {
lkp->lk_sharecount -= decr;
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
/*
Convert all simplelocks to mutexes and remove the simplelock implementations.
2001-01-24 12:35:55 +00:00
* This is the waitloop optimization.
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
*/
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
static int
Correct a locking error in apause: It should always hold the simple lock when it returns. Also, eliminate spinning on a uniprocessor. It's pointless. Submitted by: bde, Assar Westerlund <assar@sics.se>
1999-11-11 03:02:03 +00:00
apause(struct lock *lkp, int flags)
{
#ifdef SMP
int i, lock_wait;
#endif
if ((lkp->lk_flags & flags) == 0)
return 0;
#ifdef SMP
for (lock_wait = LOCK_WAIT_TIME; lock_wait > 0; lock_wait--) {
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Correct a locking error in apause: It should always hold the simple lock when it returns. Also, eliminate spinning on a uniprocessor. It's pointless. Submitted by: bde, Assar Westerlund <assar@sics.se>
1999-11-11 03:02:03 +00:00
for (i = LOCK_SAMPLE_WAIT; i > 0; i--)
if ((lkp->lk_flags & flags) == 0)
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
break;
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(lkp->lk_interlock);
Correct a locking error in apause: It should always hold the simple lock when it returns. Also, eliminate spinning on a uniprocessor. It's pointless. Submitted by: bde, Assar Westerlund <assar@sics.se>
1999-11-11 03:02:03 +00:00
if ((lkp->lk_flags & flags) == 0)
return 0;
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
Correct a locking error in apause: It should always hold the simple lock when it returns. Also, eliminate spinning on a uniprocessor. It's pointless. Submitted by: bde, Assar Westerlund <assar@sics.se>
1999-11-11 03:02:03 +00:00
#endif
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
return 1;
}
static int
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
acquire(struct lock **lkpp, int extflags, int wanted) {
struct lock *lkp = *lkpp;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
int s, error;
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
CTR3(KTR_LOCKMGR,
"acquire(): lkp == %p, extflags == 0x%x, wanted == 0x%x\n",
lkp, extflags, wanted);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if ((extflags & LK_NOWAIT) && (lkp->lk_flags & wanted)) {
return EBUSY;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
}
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
if (((lkp->lk_flags | extflags) & LK_NOPAUSE) == 0) {
error = apause(lkp, wanted);
if (error == 0)
return 0;
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
s = splhigh();
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
while ((lkp->lk_flags & wanted) != 0) {
lkp->lk_flags |= LK_WAIT_NONZERO;
lkp->lk_waitcount++;
Use msleep instead of mtx_exit; tsleep; mtx_enter, which is not safe.
2000-12-01 02:18:38 +00:00
error = msleep(lkp, lkp->lk_interlock, lkp->lk_prio,
Fix a BUF_TIMELOCK race against BUF_LOCK and fix a deadlock in vget() against VM_WAIT in the pageout code. Both fixes involve adjusting the lockmgr's timeout capability so locks obtained with timeouts do not interfere with locks obtained without a timeout. Hopefully MFC: before the 4.5 release
2001-12-20 22:42:27 +00:00
lkp->lk_wmesg,
((extflags & LK_TIMELOCK) ? lkp->lk_timo : 0));
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
if (lkp->lk_waitcount == 1) {
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
lkp->lk_flags &= ~LK_WAIT_NONZERO;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
lkp->lk_waitcount = 0;
} else {
lkp->lk_waitcount--;
}
if (error) {
splx(s);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
return error;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if (extflags & LK_SLEEPFAIL) {
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
splx(s);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
return ENOLCK;
}
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
if (lkp->lk_newlock != NULL) {
mtx_lock(lkp->lk_newlock->lk_interlock);
mtx_unlock(lkp->lk_interlock);
if (lkp->lk_waitcount == 0)
wakeup((void *)(&lkp->lk_newlock));
*lkpp = lkp = lkp->lk_newlock;
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
splx(s);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
return 0;
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Set, change, or release a lock.
*
* Shared requests increment the shared count. Exclusive requests set the
* LK_WANT_EXCL flag (preventing further shared locks), and wait for already
* accepted shared locks and shared-to-exclusive upgrades to go away.
*/
int
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#ifndef DEBUG_LOCKS
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
lockmgr(lkp, flags, interlkp, td)
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#else
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
debuglockmgr(lkp, flags, interlkp, td, name, file, line)
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#endif
pushed down "volatility" of simplelock to actual int inside the struct. Submitted by: bde@zeta.org.au, smp@csn.net
1997-08-04 19:11:26 +00:00
struct lock *lkp;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
u_int flags;
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
struct mtx *interlkp;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct thread *td;
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#ifdef DEBUG_LOCKS
const char *name; /* Name of lock function */
const char *file; /* Name of file call is from */
int line; /* Line number in file */
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
{
int error;
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
struct thread *thr;
Proc locking.
2001-02-09 16:27:41 +00:00
int extflags, lockflags;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
CTR5(KTR_LOCKMGR,
"lockmgr(): lkp == %p (lk_wmesg == \"%s\"), flags == 0x%x, "
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
"interlkp == %p, td == %p", lkp, lkp->lk_wmesg, flags, interlkp, td);
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
error = 0;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
if (td == NULL)
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
thr = LK_KERNPROC;
Allow lockmgr to work without a current process. Disallowing that was a mistake in the lockmgr rewrite.
1997-08-19 00:27:07 +00:00
else
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
thr = td;
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
- Add an interlock argument to BUF_LOCK and BUF_TIMELOCK. - Remove the buftimelock mutex and acquire the buf's interlock to protect these fields instead. - Hold the vnode interlock while locking bufs on the clean/dirty queues. This reduces some cases from one BUF_LOCK with a LK_NOWAIT and another BUF_LOCK with a LK_TIMEFAIL to a single lock. Reviewed by: arch, mckusick
2003-02-25 03:37:48 +00:00
if ((flags & LK_INTERNAL) == 0)
mtx_lock(lkp->lk_interlock);
Assert that when using an interlock mutex it is not recursed when lockmgr() is called. Ok'd by: jhb
2001-04-20 22:38:40 +00:00
if (flags & LK_INTERLOCK) {
Instead of asserting that a mutex is not still locked after unlocking it, assert that the mutex is owned and not recursed prior to unlocking it. This should give a clearer diagnostic when a programming error is caught.
2001-04-28 12:11:01 +00:00
mtx_assert(interlkp, MA_OWNED | MA_NOTRECURSED);
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(interlkp);
Assert that when using an interlock mutex it is not recursed when lockmgr() is called. Ok'd by: jhb
2001-04-20 22:38:40 +00:00
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
- Add an interlock argument to BUF_LOCK and BUF_TIMELOCK. - Remove the buftimelock mutex and acquire the buf's interlock to protect these fields instead. - Hold the vnode interlock while locking bufs on the clean/dirty queues. This reduces some cases from one BUF_LOCK with a LK_NOWAIT and another BUF_LOCK with a LK_TIMEFAIL to a single lock. Reviewed by: arch, mckusick
2003-02-25 03:37:48 +00:00
if ((flags & (LK_NOWAIT|LK_RELEASE)) == 0)
WITNESS_SLEEP(1, &lkp->lk_interlock->mtx_object);
If we've panic'd already, then just bail in lockmgr rather than blocking or possibly panic'ing again.
2001-08-10 23:29:15 +00:00
if (panicstr != NULL) {
mtx_unlock(lkp->lk_interlock);
return (0);
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
extflags = (flags | lkp->lk_flags) & LK_EXTFLG_MASK;
switch (flags & LK_TYPE_MASK) {
case LK_SHARED:
This solves a deadlock that can occur when read()ing into a file-mmap() space. When doing this, it is possible to for another process to attempt to get an exclusive lock on the vnode and deadlock the mmap/read combination when the uiomove() call tries to obtain a second shared lock on the vnode. There is still a potential deadlock situation with write()/mmap(). Submitted by: Matt Dillon <dillon@freebsd.org> Reviewed by: Luoqi Chen <luoqi@freebsd.org> Delimmitted by tag PRE_MATT_MMAP_LOCK and POST_MATT_MMAP_LOCK in kern/kern_lock.c kern/kern_subr.c
1999-03-12 03:09:29 +00:00
/*
* If we are not the exclusive lock holder, we have to block
* while there is an exclusive lock holder or while an
* exclusive lock request or upgrade request is in progress.
*
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
* However, if TDF_DEADLKTREAT is set, we override exclusive
This solves a deadlock that can occur when read()ing into a file-mmap() space. When doing this, it is possible to for another process to attempt to get an exclusive lock on the vnode and deadlock the mmap/read combination when the uiomove() call tries to obtain a second shared lock on the vnode. There is still a potential deadlock situation with write()/mmap(). Submitted by: Matt Dillon <dillon@freebsd.org> Reviewed by: Luoqi Chen <luoqi@freebsd.org> Delimmitted by tag PRE_MATT_MMAP_LOCK and POST_MATT_MMAP_LOCK in kern/kern_lock.c kern/kern_subr.c
1999-03-12 03:09:29 +00:00
* lock requests or upgrade requests ( but not the exclusive
* lock itself ).
*/
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder != thr) {
Proc locking.
2001-02-09 16:27:41 +00:00
lockflags = LK_HAVE_EXCL;
Fix locking on td_flags for TDF_DEADLKTREAT. If the comments in the code are true that curthread can change during this function, then this flag needs to become a KSE flag, not a thread flag.
2001-09-13 22:33:37 +00:00
mtx_lock_spin(&sched_lock);
if (td != NULL && !(td->td_flags & TDF_DEADLKTREAT))
lockflags |= LK_WANT_EXCL | LK_WANT_UPGRADE;
mtx_unlock_spin(&sched_lock);
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
error = acquire(&lkp, extflags, lockflags);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (error)
break;
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
sharelock(lkp, 1);
Record the file, line, and pid of the last successful shared lock holder. This is useful as a last effort in debugging file system deadlocks. This is enabled via 'options DEBUG_LOCKS'
2002-05-30 05:55:22 +00:00
#if defined(DEBUG_LOCKS)
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
lkp->lk_slockholder = thr;
Record the file, line, and pid of the last successful shared lock holder. This is useful as a last effort in debugging file system deadlocks. This is enabled via 'options DEBUG_LOCKS'
2002-05-30 05:55:22 +00:00
lkp->lk_sfilename = file;
lkp->lk_slineno = line;
lkp->lk_slockername = name;
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
}
/*
* We hold an exclusive lock, so downgrade it to shared.
* An alternative would be to fail with EDEADLK.
*/
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
sharelock(lkp, 1);
Replace various spelling with FALLTHROUGH which is lint()able
2002-08-25 13:23:09 +00:00
/* FALLTHROUGH downgrade */
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
case LK_DOWNGRADE:
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
KASSERT(lkp->lk_lockholder == thr && lkp->lk_exclusivecount != 0,
convert if/panic -> KASSERT, explain what triggered the assertion
2001-04-13 10:15:53 +00:00
("lockmgr: not holding exclusive lock "
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
"(owner thread (%p) != thread (%p), exlcnt (%d) != 0",
lkp->lk_lockholder, thr, lkp->lk_exclusivecount));
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
sharelock(lkp, lkp->lk_exclusivecount);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_exclusivecount = 0;
lkp->lk_flags &= ~LK_HAVE_EXCL;
Reversion of commit by Davidxu plus fixes since applied. I'm not convinced there is anything major wrong with the patch but them's the rules.. I am using my "David's mentor" hat to revert this as he's offline for a while.
2003-02-01 12:17:09 +00:00
lkp->lk_lockholder = LK_NOPROC;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (lkp->lk_waitcount)
wakeup((void *)lkp);
break;
case LK_EXCLUPGRADE:
/*
* If another process is ahead of us to get an upgrade,
* then we want to fail rather than have an intervening
* exclusive access.
*/
if (lkp->lk_flags & LK_WANT_UPGRADE) {
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
shareunlock(lkp, 1);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
error = EBUSY;
break;
}
Replace various spelling with FALLTHROUGH which is lint()able
2002-08-25 13:23:09 +00:00
/* FALLTHROUGH normal upgrade */
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
case LK_UPGRADE:
/*
* Upgrade a shared lock to an exclusive one. If another
* shared lock has already requested an upgrade to an
* exclusive lock, our shared lock is released and an
* exclusive lock is requested (which will be granted
* after the upgrade). If we return an error, the file
* will always be unlocked.
*/
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if ((lkp->lk_lockholder == thr) || (lkp->lk_sharecount <= 0))
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: upgrade exclusive lock");
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
shareunlock(lkp, 1);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* If we are just polling, check to see if we will block.
*/
if ((extflags & LK_NOWAIT) &&
((lkp->lk_flags & LK_WANT_UPGRADE) ||
lkp->lk_sharecount > 1)) {
error = EBUSY;
break;
}
if ((lkp->lk_flags & LK_WANT_UPGRADE) == 0) {
/*
* We are first shared lock to request an upgrade, so
* request upgrade and wait for the shared count to
* drop to zero, then take exclusive lock.
*/
lkp->lk_flags |= LK_WANT_UPGRADE;
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
error = acquire(&lkp, extflags, LK_SHARE_NONZERO);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_WANT_UPGRADE;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (error)
break;
lkp->lk_flags |= LK_HAVE_EXCL;
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
lkp->lk_lockholder = thr;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (lkp->lk_exclusivecount != 0)
panic("lockmgr: non-zero exclusive count");
lkp->lk_exclusivecount = 1;
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#if defined(DEBUG_LOCKS)
lkp->lk_filename = file;
lkp->lk_lineno = line;
lkp->lk_lockername = name;
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
}
/*
* Someone else has requested upgrade. Release our shared
* lock, awaken upgrade requestor if we are the last shared
* lock, then request an exclusive lock.
*/
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if ( (lkp->lk_flags & (LK_SHARE_NONZERO|LK_WAIT_NONZERO)) ==
LK_WAIT_NONZERO)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
wakeup((void *)lkp);
Replace various spelling with FALLTHROUGH which is lint()able
2002-08-25 13:23:09 +00:00
/* FALLTHROUGH exclusive request */
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
case LK_EXCLUSIVE:
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder == thr && thr != LK_KERNPROC) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Recursive lock.
*/
When requesting an exclusive lock with LK_NOWAIT, do not panic if LK_RECURSIVE is not set, as we will simply return that the lock is busy and not actually deadlock. This allows processes to use polling locks against buffers that they may already hold exclusively locked.
1999-06-28 07:54:58 +00:00
if ((extflags & (LK_NOWAIT | LK_CANRECURSE)) == 0)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: locking against myself");
When requesting an exclusive lock with LK_NOWAIT, do not panic if LK_RECURSIVE is not set, as we will simply return that the lock is busy and not actually deadlock. This allows processes to use polling locks against buffers that they may already hold exclusively locked.
1999-06-28 07:54:58 +00:00
if ((extflags & LK_CANRECURSE) != 0) {
lkp->lk_exclusivecount++;
break;
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
}
/*
* If we are just polling, check to see if we will sleep.
*/
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if ((extflags & LK_NOWAIT) &&
(lkp->lk_flags & (LK_HAVE_EXCL | LK_WANT_EXCL | LK_WANT_UPGRADE | LK_SHARE_NONZERO))) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
error = EBUSY;
break;
}
/*
* Try to acquire the want_exclusive flag.
*/
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
error = acquire(&lkp, extflags, (LK_HAVE_EXCL | LK_WANT_EXCL));
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (error)
break;
lkp->lk_flags |= LK_WANT_EXCL;
/*
* Wait for shared locks and upgrades to finish.
*/
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
error = acquire(&lkp, extflags, LK_WANT_UPGRADE | LK_SHARE_NONZERO);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_WANT_EXCL;
if (error)
break;
lkp->lk_flags |= LK_HAVE_EXCL;
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
lkp->lk_lockholder = thr;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (lkp->lk_exclusivecount != 0)
panic("lockmgr: non-zero exclusive count");
lkp->lk_exclusivecount = 1;
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#if defined(DEBUG_LOCKS)
lkp->lk_filename = file;
lkp->lk_lineno = line;
lkp->lk_lockername = name;
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
case LK_RELEASE:
if (lkp->lk_exclusivecount != 0) {
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder != thr &&
Fix process p_locks accounting. Conversions of the owner to LK_KERNPROC caused p_locks to be improperly accounted. Submitted by: Tor.Egge@fast.no
1999-09-27 00:21:43 +00:00
lkp->lk_lockholder != LK_KERNPROC) {
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
panic("lockmgr: thread %p, not %s %p unlocking",
thr, "exclusive lock holder",
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_lockholder);
Fix process p_locks accounting. Conversions of the owner to LK_KERNPROC caused p_locks to be improperly accounted. Submitted by: Tor.Egge@fast.no
1999-09-27 00:21:43 +00:00
}
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
if (lkp->lk_exclusivecount == 1) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_HAVE_EXCL;
lkp->lk_lockholder = LK_NOPROC;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
lkp->lk_exclusivecount = 0;
} else {
lkp->lk_exclusivecount--;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
}
Convert all simplelocks to mutexes and remove the simplelock implementations.
2001-01-24 12:35:55 +00:00
} else if (lkp->lk_flags & LK_SHARE_NONZERO)
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
shareunlock(lkp, 1);
if (lkp->lk_flags & LK_WAIT_NONZERO)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
wakeup((void *)lkp);
break;
case LK_DRAIN:
/*
* Check that we do not already hold the lock, as it can
* never drain if we do. Unfortunately, we have no way to
* check for holding a shared lock, but at least we can
* check for an exclusive one.
*/
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder == thr)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: draining against myself");
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
error = acquiredrain(lkp, extflags);
if (error)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
lkp->lk_flags |= LK_DRAINING | LK_HAVE_EXCL;
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
lkp->lk_lockholder = thr;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_exclusivecount = 1;
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#if defined(DEBUG_LOCKS)
lkp->lk_filename = file;
lkp->lk_lineno = line;
lkp->lk_lockername = name;
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
default:
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: unknown locktype request %d",
flags & LK_TYPE_MASK);
/* NOTREACHED */
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if ((lkp->lk_flags & LK_WAITDRAIN) &&
(lkp->lk_flags & (LK_HAVE_EXCL | LK_WANT_EXCL | LK_WANT_UPGRADE |
LK_SHARE_NONZERO | LK_WAIT_NONZERO)) == 0) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_WAITDRAIN;
wakeup((void *)&lkp->lk_flags);
}
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
return (error);
}
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
static int
acquiredrain(struct lock *lkp, int extflags) {
int error;
if ((extflags & LK_NOWAIT) && (lkp->lk_flags & LK_ALL)) {
return EBUSY;
}
error = apause(lkp, LK_ALL);
if (error == 0)
return 0;
while (lkp->lk_flags & LK_ALL) {
lkp->lk_flags |= LK_WAITDRAIN;
Use msleep instead of mtx_exit; tsleep; mtx_enter, which is not safe.
2000-12-01 02:18:38 +00:00
error = msleep(&lkp->lk_flags, lkp->lk_interlock, lkp->lk_prio,
Fix a BUF_TIMELOCK race against BUF_LOCK and fix a deadlock in vget() against VM_WAIT in the pageout code. Both fixes involve adjusting the lockmgr's timeout capability so locks obtained with timeouts do not interfere with locks obtained without a timeout. Hopefully MFC: before the 4.5 release
2001-12-20 22:42:27 +00:00
lkp->lk_wmesg,
((extflags & LK_TIMELOCK) ? lkp->lk_timo : 0));
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
if (error)
return error;
if (extflags & LK_SLEEPFAIL) {
return ENOLCK;
}
}
return 0;
}
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
/*
* Transfer any waiting processes from one lock to another.
*/
void
transferlockers(from, to)
struct lock *from;
struct lock *to;
{
KASSERT(from != to, ("lock transfer to self"));
KASSERT((from->lk_flags&LK_WAITDRAIN) == 0, ("transfer draining lock"));
if (from->lk_waitcount == 0)
return;
from->lk_newlock = to;
wakeup((void *)from);
msleep(&from->lk_newlock, NULL, from->lk_prio, "lkxfer", 0);
from->lk_newlock = NULL;
from->lk_flags &= ~(LK_WANT_EXCL | LK_WANT_UPGRADE);
KASSERT(from->lk_waitcount == 0, ("active lock"));
}
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
/*
* Initialize a lock; required before use.
*/
void
lockinit(lkp, prio, wmesg, timo, flags)
struct lock *lkp;
int prio;
Change wmesg to const char * instead of char *
2002-03-05 17:45:12 +00:00
const char *wmesg;
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
int timo;
int flags;
{
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
CTR5(KTR_LOCKMGR, "lockinit(): lkp == %p, prio == %d, wmesg == \"%s\", "
"timo == %d, flags = 0x%x\n", lkp, prio, wmesg, timo, flags);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
Create a mutex pool API for short term leaf mutexes. Replace the manual mutex pool in kern_lock.c (lockmgr locks) with the new API. Replace the mutexes embedded in sxlocks with the new API.
2001-11-13 21:55:13 +00:00
if (lock_mtx_valid == 0) {
Change callers of mtx_init() to pass in an appropriate lock type name. In most cases NULL is passed, but in some cases such as network driver locks (which use the MTX_NETWORK_LOCK macro) and UMA zone locks, a name is used. Tested on: i386, alpha, sparc64
2002-04-04 21:03:38 +00:00
mtx_init(&lock_mtx, "lockmgr", NULL, MTX_DEF);
Create a mutex pool API for short term leaf mutexes. Replace the manual mutex pool in kern_lock.c (lockmgr locks) with the new API. Replace the mutexes embedded in sxlocks with the new API.
2001-11-13 21:55:13 +00:00
lock_mtx_valid = 1;
}
/*
* XXX cleanup - make sure mtxpool is always initialized before
* this is ever called.
*/
if (mtx_pool_valid) {
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(&lock_mtx);
Create a mutex pool API for short term leaf mutexes. Replace the manual mutex pool in kern_lock.c (lockmgr locks) with the new API. Replace the mutexes embedded in sxlocks with the new API.
2001-11-13 21:55:13 +00:00
lkp->lk_interlock = mtx_pool_alloc();
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&lock_mtx);
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
} else {
lkp->lk_interlock = &lock_mtx;
}
lkp->lk_flags = (flags & LK_EXTFLG_MASK);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
lkp->lk_sharecount = 0;
lkp->lk_waitcount = 0;
lkp->lk_exclusivecount = 0;
lkp->lk_prio = prio;
lkp->lk_wmesg = wmesg;
lkp->lk_timo = timo;
lkp->lk_lockholder = LK_NOPROC;
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
lkp->lk_newlock = NULL;
Have lockinit() initialize the debugging fields of a lock when DEBUG_LOCKS is defined. Sponsored by: DARPA & NAI Labs.
2002-10-18 01:34:10 +00:00
#ifdef DEBUG_LOCKS
lkp->lk_filename = "none";
lkp->lk_lockername = "never exclusive locked";
lkp->lk_lineno = 0;
lkp->lk_slockholder = LK_NOPROC;
lkp->lk_sfilename = "none";
lkp->lk_slockername = "never share locked";
lkp->lk_slineno = 0;
#endif
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
}
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
/*
* Destroy a lock.
*/
void
lockdestroy(lkp)
struct lock *lkp;
{
CTR2(KTR_LOCKMGR, "lockdestroy(): lkp == %p (lk_wmesg == \"%s\")",
lkp, lkp->lk_wmesg);
}
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
/*
* Determine the status of a lock.
*/
int
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
lockstatus(lkp, td)
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
struct lock *lkp;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct thread *td;
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
{
int lock_type = 0;
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(lkp->lk_interlock);
Lock reporting and assertion changes. * lockstatus() and VOP_ISLOCKED() gets a new process argument and a new return value: LK_EXCLOTHER, when the lock is held exclusively by another process. * The ASSERT_VOP_(UN)LOCKED family is extended to use what this gives them * Extend the vnode_if.src format to allow more exact specification than locked/unlocked. This commit should not do any semantic changes unless you are using DEBUG_VFS_LOCKS. Discussed with: grog, mch, peter, phk Reviewed by: peter
1999-12-11 16:13:02 +00:00
if (lkp->lk_exclusivecount != 0) {
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (td == NULL || lkp->lk_lockholder == td)
Lock reporting and assertion changes. * lockstatus() and VOP_ISLOCKED() gets a new process argument and a new return value: LK_EXCLOTHER, when the lock is held exclusively by another process. * The ASSERT_VOP_(UN)LOCKED family is extended to use what this gives them * Extend the vnode_if.src format to allow more exact specification than locked/unlocked. This commit should not do any semantic changes unless you are using DEBUG_VFS_LOCKS. Discussed with: grog, mch, peter, phk Reviewed by: peter
1999-12-11 16:13:02 +00:00
lock_type = LK_EXCLUSIVE;
else
lock_type = LK_EXCLOTHER;
} else if (lkp->lk_sharecount != 0)
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
lock_type = LK_SHARED;
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
return (lock_type);
}
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
/*
* Determine the number of holders of a lock.
*/
int
lockcount(lkp)
struct lock *lkp;
{
int count;
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(lkp->lk_interlock);
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
count = lkp->lk_exclusivecount + lkp->lk_sharecount;
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
return (count);
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Print out information about state of a lock. Used by VOP_PRINT
typo in comment.
1997-08-22 07:16:46 +00:00
* routines to display status about contained locks.
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
*/
Replace original revision 1.1; Author dyson; Date: 1997/02/10 02:28:15 Changes from Lite2: - DEBUG -> SIMPLELOCK_DEBUG - cosmetic fixes - bzero of lock at init time -> explicit init of members.
1997-03-25 16:32:46 +00:00
void
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lockmgr_printinfo(lkp)
struct lock *lkp;
{
if (lkp->lk_sharecount)
printf(" lock type %s: SHARED (count %d)", lkp->lk_wmesg,
lkp->lk_sharecount);
else if (lkp->lk_flags & LK_HAVE_EXCL)
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
printf(" lock type %s: EXCL (count %d) by thread %p",
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_wmesg, lkp->lk_exclusivecount, lkp->lk_lockholder);
if (lkp->lk_waitcount > 0)
printf(" with %d pending", lkp->lk_waitcount);
}
Reference in New Issue Copy Permalink