d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
54f0b44b6e
freebsd-skq/sys/kern/kern_lock.c

686 lines
18 KiB
C
Raw Normal View History

/* -> /*- for copyright notices, minor format tweaks as necessary
2005-01-06 23:35:40 +00:00
/*-
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
* Copyright (c) 1995
* The Regents of the University of California. All rights reserved.
*
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
* Copyright (C) 1997
* John S. Dyson. All rights reserved.
*
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
* This code contains ideas from software contributed to Berkeley by
* Avadis Tevanian, Jr., Michael Wayne Young, and the Mach Operating
* System project at Carnegie-Mellon University.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. All advertising materials mentioning features or use of this software
* must display the following acknowledgement:
* This product includes software developed by the University of
* California, Berkeley and its contributors.
* 4. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)kern_lock.c 8.18 (Berkeley) 5/21/95
*/
Use __FBSDID().
2003-06-11 00:56:59 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
Add a 'show lockmgr' command that dumps the relevant details of a lockmgr lock.
2006-08-15 16:42:16 +00:00
#include "opt_ddb.h"
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
#include "opt_global.h"
Add a 'show lockmgr' command that dumps the relevant details of a lockmgr lock.
2006-08-15 16:42:16 +00:00
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
#include <sys/param.h>
Include kdb.h so that kdb_active is declared regardless of KDB being included in the kernel. MFC after: 0 days
2005-10-02 10:03:51 +00:00
#include <sys/kdb.h>
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
#include <sys/kernel.h>
Add missing includes of sys/ktr.h.
2001-10-11 17:53:43 +00:00
#include <sys/ktr.h>
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
#include <sys/lock.h>
Include <sys/lockmgr.h> for the definitions of the locking interfaces that are implemented here instead of depending on namespace pollution in <sys/lock.h>. Fixed nearby include messes (1 disordered include and 1 unused include).
2002-08-27 09:59:47 +00:00
#include <sys/lockmgr.h>
- machine/mutex.h -> sys/mutex.h - The initial lock_mtx mutex used in the lockmgr code is initialized very early, so use MUTEX_DECLARE() and MTX_COLD.
2000-10-20 07:28:00 +00:00
#include <sys/mutex.h>
Include <sys/lockmgr.h> for the definitions of the locking interfaces that are implemented here instead of depending on namespace pollution in <sys/lock.h>. Fixed nearby include messes (1 disordered include and 1 unused include).
2002-08-27 09:59:47 +00:00
#include <sys/proc.h>
Replace original rev 1.2; Author: mpp; Date: 1997/02/12 06:52:30 Add missing #include <sys/systm.h>
1997-03-25 16:36:35 +00:00
#include <sys/systm.h>
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
#include <sys/lock_profile.h>
- Replace the series of DEBUG_LOCKS hacks which tried to save the vn_lock caller by saving the stack of the last locker/unlocker in lockmgr. We also put the stack in KTR at the moment. Contributed by: Antoine Brodin <antoine.brodin@laposte.net>
2005-08-03 04:48:22 +00:00
#ifdef DEBUG_LOCKS
#include <sys/stack.h>
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
Add a 'show lockmgr' command that dumps the relevant details of a lockmgr lock.
2006-08-15 16:42:16 +00:00
#ifdef DDB
#include <ddb/ddb.h>
track lock class name in a way that doesn't break WITNESS
2006-11-13 05:41:46 +00:00
static void db_show_lockmgr(struct lock_object *lock);
Add a 'show lockmgr' command that dumps the relevant details of a lockmgr lock.
2006-08-15 16:42:16 +00:00
#endif
track lock class name in a way that doesn't break WITNESS
2006-11-13 05:41:46 +00:00
struct lock_class lock_class_lockmgr = {
"lockmgr",
LC_SLEEPLOCK | LC_SLEEPABLE | LC_RECURSABLE | LC_UPGRADABLE,
#ifdef DDB
db_show_lockmgr
#endif
};
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Locking primitives implementation.
* Locks provide shared/exclusive sychronization.
*/
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
#define COUNT(td, x) if ((td)) (td)->td_locks += (x)
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
#define LK_ALL (LK_HAVE_EXCL | LK_WANT_EXCL | LK_WANT_UPGRADE | \
LK_SHARE_NONZERO | LK_WAIT_NONZERO)
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
static int acquire(struct lock **lkpp, int extflags, int wanted, int *contested, uint64_t *waittime);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
static int acquiredrain(struct lock *lkp, int extflags) ;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
- Remove dead code.
2005-04-06 10:11:14 +00:00
static __inline void
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
sharelock(struct thread *td, struct lock *lkp, int incr) {
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
lkp->lk_flags |= LK_SHARE_NONZERO;
lkp->lk_sharecount += incr;
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
COUNT(td, incr);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
- Remove dead code.
2005-04-06 10:11:14 +00:00
static __inline void
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
shareunlock(struct thread *td, struct lock *lkp, int decr) {
KNFize, by bde.
1999-01-10 01:58:29 +00:00
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
KASSERT(lkp->lk_sharecount >= decr, ("shareunlock: count < decr"));
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
COUNT(td, -decr);
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
if (lkp->lk_sharecount == decr) {
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
lkp->lk_flags &= ~LK_SHARE_NONZERO;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
if (lkp->lk_flags & (LK_WANT_UPGRADE | LK_WANT_EXCL)) {
wakeup(lkp);
}
lkp->lk_sharecount = 0;
} else {
lkp->lk_sharecount -= decr;
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
static int
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
acquire(struct lock **lkpp, int extflags, int wanted, int *contested, uint64_t *waittime)
- Do not use APAUSE if LK_INTERLOCK is set. We lose synchronization if the lockmgr interlock is dropped after the caller's interlock is dropped. - Change some lockmgr KTRs to be slightly more helpful. Sponsored By: Isilon Systems, Inc.
2005-01-24 10:20:59 +00:00
{
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
struct lock *lkp = *lkpp;
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
int error;
Use the KTR_LOCK mask for logging events via KTR in lockmgr() rather than KTR_LOCKMGR. lockmgr locks are locks just like other locks.
2003-03-11 20:00:37 +00:00
CTR3(KTR_LOCK,
Don't include a "\n" in KTR output, it confuses automatic parsing.
2004-07-23 20:12:56 +00:00
"acquire(): lkp == %p, extflags == 0x%x, wanted == 0x%x",
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
lkp, extflags, wanted);
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
if ((extflags & LK_NOWAIT) && (lkp->lk_flags & wanted))
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
return EBUSY;
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
error = 0;
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
if ((lkp->lk_flags & wanted) != 0)
lock_profile_obtain_lock_failed(&lkp->lk_object, contested, waittime);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
while ((lkp->lk_flags & wanted) != 0) {
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
CTR2(KTR_LOCK,
"acquire(): lkp == %p, lk_flags == 0x%x sleeping",
lkp, lkp->lk_flags);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
lkp->lk_flags |= LK_WAIT_NONZERO;
lkp->lk_waitcount++;
Use msleep instead of mtx_exit; tsleep; mtx_enter, which is not safe.
2000-12-01 02:18:38 +00:00
error = msleep(lkp, lkp->lk_interlock, lkp->lk_prio,
Fix a BUF_TIMELOCK race against BUF_LOCK and fix a deadlock in vget() against VM_WAIT in the pageout code. Both fixes involve adjusting the lockmgr's timeout capability so locks obtained with timeouts do not interfere with locks obtained without a timeout. Hopefully MFC: before the 4.5 release
2001-12-20 22:42:27 +00:00
lkp->lk_wmesg,
((extflags & LK_TIMELOCK) ? lkp->lk_timo : 0));
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
lkp->lk_waitcount--;
if (lkp->lk_waitcount == 0)
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
lkp->lk_flags &= ~LK_WAIT_NONZERO;
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
if (error)
break;
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if (extflags & LK_SLEEPFAIL) {
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
error = ENOLCK;
break;
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
if (lkp->lk_newlock != NULL) {
mtx_lock(lkp->lk_newlock->lk_interlock);
mtx_unlock(lkp->lk_interlock);
if (lkp->lk_waitcount == 0)
wakeup((void *)(&lkp->lk_newlock));
*lkpp = lkp = lkp->lk_newlock;
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
- Slightly restructure acquire() so I can add more ktr information and an assert to help find two strange bugs. - Remove some nearby spls.
2005-04-03 11:49:02 +00:00
mtx_assert(lkp->lk_interlock, MA_OWNED);
return (error);
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Set, change, or release a lock.
*
* Shared requests increment the shared count. Exclusive requests set the
* LK_WANT_EXCL flag (preventing further shared locks), and wait for already
* accepted shared locks and shared-to-exclusive upgrades to go away.
*/
int
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
_lockmgr(struct lock *lkp, int flags, struct mtx *interlkp,
struct thread *td, char *file, int line)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
{
int error;
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
struct thread *thr;
Proc locking.
2001-02-09 16:27:41 +00:00
int extflags, lockflags;
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
int contested = 0;
uint64_t waitstart = 0;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
error = 0;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
if (td == NULL)
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
thr = LK_KERNPROC;
Allow lockmgr to work without a current process. Disallowing that was a mistake in the lockmgr rewrite.
1997-08-19 00:27:07 +00:00
else
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
thr = td;
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
- Add an interlock argument to BUF_LOCK and BUF_TIMELOCK. - Remove the buftimelock mutex and acquire the buf's interlock to protect these fields instead. - Hold the vnode interlock while locking bufs on the clean/dirty queues. This reduces some cases from one BUF_LOCK with a LK_NOWAIT and another BUF_LOCK with a LK_TIMEFAIL to a single lock. Reviewed by: arch, mckusick
2003-02-25 03:37:48 +00:00
if ((flags & LK_INTERNAL) == 0)
mtx_lock(lkp->lk_interlock);
- Do not use APAUSE if LK_INTERLOCK is set. We lose synchronization if the lockmgr interlock is dropped after the caller's interlock is dropped. - Change some lockmgr KTRs to be slightly more helpful. Sponsored By: Isilon Systems, Inc.
2005-01-24 10:20:59 +00:00
CTR6(KTR_LOCK,
"lockmgr(): lkp == %p (lk_wmesg == \"%s\"), owner == %p, exclusivecount == %d, flags == 0x%x, "
"td == %p", lkp, lkp->lk_wmesg, lkp->lk_lockholder,
lkp->lk_exclusivecount, flags, td);
- Replace the series of DEBUG_LOCKS hacks which tried to save the vn_lock caller by saving the stack of the last locker/unlocker in lockmgr. We also put the stack in KTR at the moment. Contributed by: Antoine Brodin <antoine.brodin@laposte.net>
2005-08-03 04:48:22 +00:00
#ifdef DEBUG_LOCKS
{
struct stack stack; /* XXX */
stack_save(&stack);
Add 'depth' argument to CTRSTACK() macro, which allows to reduce number of ktr slots used. If 'depth' is equal to 0, the whole stack will be logged, just like before.
2005-08-29 11:34:08 +00:00
CTRSTACK(KTR_LOCK, &stack, 0, 1);
- Replace the series of DEBUG_LOCKS hacks which tried to save the vn_lock caller by saving the stack of the last locker/unlocker in lockmgr. We also put the stack in KTR at the moment. Contributed by: Antoine Brodin <antoine.brodin@laposte.net>
2005-08-03 04:48:22 +00:00
}
- Do not use APAUSE if LK_INTERLOCK is set. We lose synchronization if the lockmgr interlock is dropped after the caller's interlock is dropped. - Change some lockmgr KTRs to be slightly more helpful. Sponsored By: Isilon Systems, Inc.
2005-01-24 10:20:59 +00:00
#endif
Assert that when using an interlock mutex it is not recursed when lockmgr() is called. Ok'd by: jhb
2001-04-20 22:38:40 +00:00
if (flags & LK_INTERLOCK) {
Instead of asserting that a mutex is not still locked after unlocking it, assert that the mutex is owned and not recursed prior to unlocking it. This should give a clearer diagnostic when a programming error is caught.
2001-04-28 12:11:01 +00:00
mtx_assert(interlkp, MA_OWNED | MA_NOTRECURSED);
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(interlkp);
Assert that when using an interlock mutex it is not recursed when lockmgr() is called. Ok'd by: jhb
2001-04-20 22:38:40 +00:00
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
- Add an interlock argument to BUF_LOCK and BUF_TIMELOCK. - Remove the buftimelock mutex and acquire the buf's interlock to protect these fields instead. - Hold the vnode interlock while locking bufs on the clean/dirty queues. This reduces some cases from one BUF_LOCK with a LK_NOWAIT and another BUF_LOCK with a LK_TIMEFAIL to a single lock. Reviewed by: arch, mckusick
2003-02-25 03:37:48 +00:00
if ((flags & (LK_NOWAIT|LK_RELEASE)) == 0)
Replace calls to WITNESS_SLEEP() and witness_list() with equivalent calls to WITNESS_WARN().
2003-03-04 21:03:05 +00:00
WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK,
&lkp->lk_interlock->mtx_object,
"Acquiring lockmgr lock \"%s\"", lkp->lk_wmesg);
- Add an interlock argument to BUF_LOCK and BUF_TIMELOCK. - Remove the buftimelock mutex and acquire the buf's interlock to protect these fields instead. - Hold the vnode interlock while locking bufs on the clean/dirty queues. This reduces some cases from one BUF_LOCK with a LK_NOWAIT and another BUF_LOCK with a LK_TIMEFAIL to a single lock. Reviewed by: arch, mckusick
2003-02-25 03:37:48 +00:00
If we've panic'd already, then just bail in lockmgr rather than blocking or possibly panic'ing again.
2001-08-10 23:29:15 +00:00
if (panicstr != NULL) {
mtx_unlock(lkp->lk_interlock);
return (0);
}
- Add a LK_NOSHARE flag which forces all shared lock requests to be treated as exclusive lock requests. Sponsored by: Isilon Systems, Inc.
2005-03-31 05:18:19 +00:00
if ((lkp->lk_flags & LK_NOSHARE) &&
(flags & LK_TYPE_MASK) == LK_SHARED) {
flags &= ~LK_TYPE_MASK;
flags |= LK_EXCLUSIVE;
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
extflags = (flags | lkp->lk_flags) & LK_EXTFLG_MASK;
switch (flags & LK_TYPE_MASK) {
case LK_SHARED:
This solves a deadlock that can occur when read()ing into a file-mmap() space. When doing this, it is possible to for another process to attempt to get an exclusive lock on the vnode and deadlock the mmap/read combination when the uiomove() call tries to obtain a second shared lock on the vnode. There is still a potential deadlock situation with write()/mmap(). Submitted by: Matt Dillon <dillon@freebsd.org> Reviewed by: Luoqi Chen <luoqi@freebsd.org> Delimmitted by tag PRE_MATT_MMAP_LOCK and POST_MATT_MMAP_LOCK in kern/kern_lock.c kern/kern_subr.c
1999-03-12 03:09:29 +00:00
/*
* If we are not the exclusive lock holder, we have to block
* while there is an exclusive lock holder or while an
* exclusive lock request or upgrade request is in progress.
*
Move TDF_DEADLKTREAT into td_pflags (and rename it accordingly) to avoid having to acquire sched_lock when manipulating it in lockmgr(), uiomove(), and uiomove_fromphys(). Reviewed by: jhb
2004-06-03 01:47:37 +00:00
* However, if TDP_DEADLKTREAT is set, we override exclusive
This solves a deadlock that can occur when read()ing into a file-mmap() space. When doing this, it is possible to for another process to attempt to get an exclusive lock on the vnode and deadlock the mmap/read combination when the uiomove() call tries to obtain a second shared lock on the vnode. There is still a potential deadlock situation with write()/mmap(). Submitted by: Matt Dillon <dillon@freebsd.org> Reviewed by: Luoqi Chen <luoqi@freebsd.org> Delimmitted by tag PRE_MATT_MMAP_LOCK and POST_MATT_MMAP_LOCK in kern/kern_lock.c kern/kern_subr.c
1999-03-12 03:09:29 +00:00
* lock requests or upgrade requests ( but not the exclusive
* lock itself ).
*/
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder != thr) {
Proc locking.
2001-02-09 16:27:41 +00:00
lockflags = LK_HAVE_EXCL;
Move TDF_DEADLKTREAT into td_pflags (and rename it accordingly) to avoid having to acquire sched_lock when manipulating it in lockmgr(), uiomove(), and uiomove_fromphys(). Reviewed by: jhb
2004-06-03 01:47:37 +00:00
if (td != NULL && !(td->td_pflags & TDP_DEADLKTREAT))
Fix locking on td_flags for TDF_DEADLKTREAT. If the comments in the code are true that curthread can change during this function, then this flag needs to become a KSE flag, not a thread flag.
2001-09-13 22:33:37 +00:00
lockflags |= LK_WANT_EXCL | LK_WANT_UPGRADE;
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
error = acquire(&lkp, extflags, lockflags, &contested, &waitstart);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (error)
break;
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
sharelock(td, lkp, 1);
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
if (lkp->lk_sharecount == 1)
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
lock_profile_obtain_lock_success(&lkp->lk_object, contested, waitstart, file, line);
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
Record the file, line, and pid of the last successful shared lock holder. This is useful as a last effort in debugging file system deadlocks. This is enabled via 'options DEBUG_LOCKS'
2002-05-30 05:55:22 +00:00
#if defined(DEBUG_LOCKS)
- Fix a problem that slipped through review; the stack member of the lockmgr structure should have the lk_ prefix. - Add stack_print(lkp->lk_stack) to the information printed with lockmgr_printinfo().
2005-08-03 04:59:07 +00:00
stack_save(&lkp->lk_stack);
Record the file, line, and pid of the last successful shared lock holder. This is useful as a last effort in debugging file system deadlocks. This is enabled via 'options DEBUG_LOCKS'
2002-05-30 05:55:22 +00:00
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
}
/*
* We hold an exclusive lock, so downgrade it to shared.
* An alternative would be to fail with EDEADLK.
*/
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
sharelock(td, lkp, 1);
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
if (lkp->lk_sharecount == 1)
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
lock_profile_obtain_lock_success(&lkp->lk_object, contested, waitstart, file, line);
Replace various spelling with FALLTHROUGH which is lint()able
2002-08-25 13:23:09 +00:00
/* FALLTHROUGH downgrade */
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
case LK_DOWNGRADE:
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
KASSERT(lkp->lk_lockholder == thr && lkp->lk_exclusivecount != 0,
convert if/panic -> KASSERT, explain what triggered the assertion
2001-04-13 10:15:53 +00:00
("lockmgr: not holding exclusive lock "
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
"(owner thread (%p) != thread (%p), exlcnt (%d) != 0",
lkp->lk_lockholder, thr, lkp->lk_exclusivecount));
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
sharelock(td, lkp, lkp->lk_exclusivecount);
COUNT(td, -lkp->lk_exclusivecount);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_exclusivecount = 0;
lkp->lk_flags &= ~LK_HAVE_EXCL;
Reversion of commit by Davidxu plus fixes since applied. I'm not convinced there is anything major wrong with the patch but them's the rules.. I am using my "David's mentor" hat to revert this as he's offline for a while.
2003-02-01 12:17:09 +00:00
lkp->lk_lockholder = LK_NOPROC;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (lkp->lk_waitcount)
wakeup((void *)lkp);
break;
case LK_EXCLUPGRADE:
/*
* If another process is ahead of us to get an upgrade,
* then we want to fail rather than have an intervening
* exclusive access.
*/
if (lkp->lk_flags & LK_WANT_UPGRADE) {
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
shareunlock(td, lkp, 1);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
error = EBUSY;
break;
}
Replace various spelling with FALLTHROUGH which is lint()able
2002-08-25 13:23:09 +00:00
/* FALLTHROUGH normal upgrade */
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
case LK_UPGRADE:
/*
* Upgrade a shared lock to an exclusive one. If another
* shared lock has already requested an upgrade to an
* exclusive lock, our shared lock is released and an
* exclusive lock is requested (which will be granted
* after the upgrade). If we return an error, the file
* will always be unlocked.
*/
- Differentiate two UPGRADE panics so I have a better idea of what's going on here.
2005-04-12 05:43:03 +00:00
if (lkp->lk_lockholder == thr)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: upgrade exclusive lock");
- Differentiate two UPGRADE panics so I have a better idea of what's going on here.
2005-04-12 05:43:03 +00:00
if (lkp->lk_sharecount <= 0)
panic("lockmgr: upgrade without shared");
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
shareunlock(td, lkp, 1);
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
if (lkp->lk_sharecount == 0)
lock_profile_release_lock(&lkp->lk_object);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* If we are just polling, check to see if we will block.
*/
if ((extflags & LK_NOWAIT) &&
((lkp->lk_flags & LK_WANT_UPGRADE) ||
lkp->lk_sharecount > 1)) {
error = EBUSY;
break;
}
if ((lkp->lk_flags & LK_WANT_UPGRADE) == 0) {
/*
* We are first shared lock to request an upgrade, so
* request upgrade and wait for the shared count to
* drop to zero, then take exclusive lock.
*/
lkp->lk_flags |= LK_WANT_UPGRADE;
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
error = acquire(&lkp, extflags, LK_SHARE_NONZERO, &contested, &waitstart);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_WANT_UPGRADE;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
Reintroduce slightly modified patch from kern/69964. Check for LK_HAVE_EXL in both acquire invocations. MFC after: 5 days
2004-08-27 01:41:28 +00:00
if (error) {
if ((lkp->lk_flags & ( LK_WANT_EXCL | LK_WAIT_NONZERO)) == (LK_WANT_EXCL | LK_WAIT_NONZERO))
wakeup((void *)lkp);
break;
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (lkp->lk_exclusivecount != 0)
panic("lockmgr: non-zero exclusive count");
When upgrading the shared lock to an exclusive lock, if we discover that the exclusive lock is already held, then we call panic. Don't clobber internal lock state before panic'ing. This change improves debugging if this case were to happen. Submitted by: Mohan Srinivasan mohans at yahoo-inc dot com Reviewed by: rwatson
2004-11-29 22:58:32 +00:00
lkp->lk_flags |= LK_HAVE_EXCL;
lkp->lk_lockholder = thr;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_exclusivecount = 1;
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
COUNT(td, 1);
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
lock_profile_obtain_lock_success(&lkp->lk_object, contested, waitstart, file, line);
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#if defined(DEBUG_LOCKS)
- Fix a problem that slipped through review; the stack member of the lockmgr structure should have the lk_ prefix. - Add stack_print(lkp->lk_stack) to the information printed with lockmgr_printinfo().
2005-08-03 04:59:07 +00:00
stack_save(&lkp->lk_stack);
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
}
/*
* Someone else has requested upgrade. Release our shared
* lock, awaken upgrade requestor if we are the last shared
* lock, then request an exclusive lock.
*/
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if ( (lkp->lk_flags & (LK_SHARE_NONZERO|LK_WAIT_NONZERO)) ==
LK_WAIT_NONZERO)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
wakeup((void *)lkp);
Replace various spelling with FALLTHROUGH which is lint()able
2002-08-25 13:23:09 +00:00
/* FALLTHROUGH exclusive request */
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
case LK_EXCLUSIVE:
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder == thr && thr != LK_KERNPROC) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Recursive lock.
*/
When requesting an exclusive lock with LK_NOWAIT, do not panic if LK_RECURSIVE is not set, as we will simply return that the lock is busy and not actually deadlock. This allows processes to use polling locks against buffers that they may already hold exclusively locked.
1999-06-28 07:54:58 +00:00
if ((extflags & (LK_NOWAIT | LK_CANRECURSE)) == 0)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: locking against myself");
When requesting an exclusive lock with LK_NOWAIT, do not panic if LK_RECURSIVE is not set, as we will simply return that the lock is busy and not actually deadlock. This allows processes to use polling locks against buffers that they may already hold exclusively locked.
1999-06-28 07:54:58 +00:00
if ((extflags & LK_CANRECURSE) != 0) {
lkp->lk_exclusivecount++;
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
COUNT(td, 1);
When requesting an exclusive lock with LK_NOWAIT, do not panic if LK_RECURSIVE is not set, as we will simply return that the lock is busy and not actually deadlock. This allows processes to use polling locks against buffers that they may already hold exclusively locked.
1999-06-28 07:54:58 +00:00
break;
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
}
/*
* If we are just polling, check to see if we will sleep.
*/
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if ((extflags & LK_NOWAIT) &&
(lkp->lk_flags & (LK_HAVE_EXCL | LK_WANT_EXCL | LK_WANT_UPGRADE | LK_SHARE_NONZERO))) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
error = EBUSY;
break;
}
/*
* Try to acquire the want_exclusive flag.
*/
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
error = acquire(&lkp, extflags, (LK_HAVE_EXCL | LK_WANT_EXCL), &contested, &waitstart);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (error)
break;
lkp->lk_flags |= LK_WANT_EXCL;
/*
* Wait for shared locks and upgrades to finish.
*/
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
error = acquire(&lkp, extflags, LK_HAVE_EXCL | LK_WANT_UPGRADE | LK_SHARE_NONZERO, &contested, &waitstart);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_WANT_EXCL;
Reintroduce slightly modified patch from kern/69964. Check for LK_HAVE_EXL in both acquire invocations. MFC after: 5 days
2004-08-27 01:41:28 +00:00
if (error) {
if (lkp->lk_flags & LK_WAIT_NONZERO)
wakeup((void *)lkp);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
Reintroduce slightly modified patch from kern/69964. Check for LK_HAVE_EXL in both acquire invocations. MFC after: 5 days
2004-08-27 01:41:28 +00:00
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags |= LK_HAVE_EXCL;
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
lkp->lk_lockholder = thr;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (lkp->lk_exclusivecount != 0)
panic("lockmgr: non-zero exclusive count");
lkp->lk_exclusivecount = 1;
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
COUNT(td, 1);
general LOCK_PROFILING cleanup - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review
2007-02-26 08:26:44 +00:00
lock_profile_obtain_lock_success(&lkp->lk_object, contested, waitstart, file, line);
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#if defined(DEBUG_LOCKS)
- Fix a problem that slipped through review; the stack member of the lockmgr structure should have the lk_ prefix. - Add stack_print(lkp->lk_stack) to the information printed with lockmgr_printinfo().
2005-08-03 04:59:07 +00:00
stack_save(&lkp->lk_stack);
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
case LK_RELEASE:
if (lkp->lk_exclusivecount != 0) {
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder != thr &&
Fix process p_locks accounting. Conversions of the owner to LK_KERNPROC caused p_locks to be improperly accounted. Submitted by: Tor.Egge@fast.no
1999-09-27 00:21:43 +00:00
lkp->lk_lockholder != LK_KERNPROC) {
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
panic("lockmgr: thread %p, not %s %p unlocking",
thr, "exclusive lock holder",
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_lockholder);
Fix process p_locks accounting. Conversions of the owner to LK_KERNPROC caused p_locks to be improperly accounted. Submitted by: Tor.Egge@fast.no
1999-09-27 00:21:43 +00:00
}
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
if (lkp->lk_lockholder != LK_KERNPROC)
COUNT(td, -1);
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
if (lkp->lk_exclusivecount == 1) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_HAVE_EXCL;
lkp->lk_lockholder = LK_NOPROC;
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
lkp->lk_exclusivecount = 0;
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
lock_profile_release_lock(&lkp->lk_object);
Some kern_lock code improvements. Add missing wakeup, and enable disabling some diagnostics when memory or speed is at a premium.
1998-03-07 19:25:34 +00:00
} else {
lkp->lk_exclusivecount--;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
}
Convert all simplelocks to mutexes and remove the simplelock implementations.
2001-01-24 12:35:55 +00:00
} else if (lkp->lk_flags & LK_SHARE_NONZERO)
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
shareunlock(td, lkp, 1);
Print out a warning and a backtrace if we try to unlock a lockmgr that we do not hold. Glanced at by: phk MFC after: 3 days
2005-09-02 15:56:01 +00:00
else {
printf("lockmgr: thread %p unlocking unheld lock\n",
thr);
kdb_backtrace();
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if (lkp->lk_flags & LK_WAIT_NONZERO)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
wakeup((void *)lkp);
break;
case LK_DRAIN:
/*
* Check that we do not already hold the lock, as it can
* never drain if we do. Unfortunately, we have no way to
* check for holding a shared lock, but at least we can
* check for an exclusive one.
*/
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (lkp->lk_lockholder == thr)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: draining against myself");
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
error = acquiredrain(lkp, extflags);
if (error)
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
lkp->lk_flags |= LK_DRAINING | LK_HAVE_EXCL;
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
lkp->lk_lockholder = thr;
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_exclusivecount = 1;
- Restore COUNT() in all of its original glory. Don't make it dependent on DEBUG as ufs will soon grow a dependency on this count. Discussed with: bde Sponsored by: Isilon Systems, Inc.
2005-03-25 00:00:44 +00:00
COUNT(td, 1);
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#if defined(DEBUG_LOCKS)
- Fix a problem that slipped through review; the stack member of the lockmgr structure should have the lk_ prefix. - Add stack_print(lkp->lk_stack) to the information printed with lockmgr_printinfo().
2005-08-03 04:59:07 +00:00
stack_save(&lkp->lk_stack);
Add 'options DEBUG_LOCKS', which stores extra information in struct lock, and add some macros and function parameters to make sure that the information get to the point where it can be put in the lock structure. While I'm here, add DEBUG_VFS_LOCKS to LINT.
1999-01-20 14:49:12 +00:00
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
break;
default:
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
panic("lockmgr: unknown locktype request %d",
flags & LK_TYPE_MASK);
/* NOTREACHED */
}
Fix kern_lock so that it will work. Additionally, clean-up some of the VM systems usage of the kernel lock (lockmgr) code. This is a first pass implementation, and is expected to evolve as needed. The API for the lock manager code has not changed, but the underlying implementation has changed significantly. This change should not materially affect our current SMP or UP code without non-standard parameters being used.
1997-08-18 02:06:35 +00:00
if ((lkp->lk_flags & LK_WAITDRAIN) &&
(lkp->lk_flags & (LK_HAVE_EXCL | LK_WANT_EXCL | LK_WANT_UPGRADE |
LK_SHARE_NONZERO | LK_WAIT_NONZERO)) == 0) {
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lkp->lk_flags &= ~LK_WAITDRAIN;
wakeup((void *)&lkp->lk_flags);
}
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
return (error);
}
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
static int
acquiredrain(struct lock *lkp, int extflags) {
int error;
if ((extflags & LK_NOWAIT) && (lkp->lk_flags & LK_ALL)) {
return EBUSY;
}
while (lkp->lk_flags & LK_ALL) {
lkp->lk_flags |= LK_WAITDRAIN;
Use msleep instead of mtx_exit; tsleep; mtx_enter, which is not safe.
2000-12-01 02:18:38 +00:00
error = msleep(&lkp->lk_flags, lkp->lk_interlock, lkp->lk_prio,
Fix a BUF_TIMELOCK race against BUF_LOCK and fix a deadlock in vget() against VM_WAIT in the pageout code. Both fixes involve adjusting the lockmgr's timeout capability so locks obtained with timeouts do not interfere with locks obtained without a timeout. Hopefully MFC: before the 4.5 release
2001-12-20 22:42:27 +00:00
lkp->lk_wmesg,
((extflags & LK_TIMELOCK) ? lkp->lk_timo : 0));
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
if (error)
return error;
if (extflags & LK_SLEEPFAIL) {
return ENOLCK;
}
}
return 0;
}
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
/*
* Transfer any waiting processes from one lock to another.
*/
void
transferlockers(from, to)
struct lock *from;
struct lock *to;
{
KASSERT(from != to, ("lock transfer to self"));
KASSERT((from->lk_flags&LK_WAITDRAIN) == 0, ("transfer draining lock"));
- transferlockers() requires the interlock to be SMP safe. Sponsored by: Isilon Systems, Inc.
2005-03-15 09:27:45 +00:00
mtx_lock(from->lk_interlock);
if (from->lk_waitcount == 0) {
mtx_unlock(from->lk_interlock);
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
return;
- transferlockers() requires the interlock to be SMP safe. Sponsored by: Isilon Systems, Inc.
2005-03-15 09:27:45 +00:00
}
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
from->lk_newlock = to;
wakeup((void *)from);
- transferlockers() requires the interlock to be SMP safe. Sponsored by: Isilon Systems, Inc.
2005-03-15 09:27:45 +00:00
msleep(&from->lk_newlock, from->lk_interlock, from->lk_prio,
"lkxfer", 0);
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
from->lk_newlock = NULL;
from->lk_flags &= ~(LK_WANT_EXCL | LK_WANT_UPGRADE);
KASSERT(from->lk_waitcount == 0, ("active lock"));
- transferlockers() requires the interlock to be SMP safe. Sponsored by: Isilon Systems, Inc.
2005-03-15 09:27:45 +00:00
mtx_unlock(from->lk_interlock);
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
}
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
/*
* Initialize a lock; required before use.
*/
void
lockinit(lkp, prio, wmesg, timo, flags)
struct lock *lkp;
int prio;
Change wmesg to const char * instead of char *
2002-03-05 17:45:12 +00:00
const char *wmesg;
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
int timo;
int flags;
{
Use the KTR_LOCK mask for logging events via KTR in lockmgr() rather than KTR_LOCKMGR. lockmgr locks are locks just like other locks.
2003-03-11 20:00:37 +00:00
CTR5(KTR_LOCK, "lockinit(): lkp == %p, prio == %d, wmesg == \"%s\", "
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
"timo == %d, flags = 0x%x\n", lkp, prio, wmesg, timo, flags);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
Rearrange the SYSINIT order to call lockmgr_init() earlier so that the runtime lockmgr initialization code in lockinit() can be eliminated. Reviewed by: jhb
2003-07-16 01:00:39 +00:00
lkp->lk_interlock = mtx_pool_alloc(mtxpool_lockbuilder);
For lockmgr mutex protection, use an array of mutexes that are allocated and initialized during boot. This avoids bloating sizeof(struct lock). As a side effect, it is no longer necessary to enforce the assumtion that lockinit()/lockdestroy() calls are paired, so the LK_VALID flag has been removed. Idea taken from: BSD/OS.
2000-10-12 22:37:28 +00:00
lkp->lk_flags = (flags & LK_EXTFLG_MASK);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
lkp->lk_sharecount = 0;
lkp->lk_waitcount = 0;
lkp->lk_exclusivecount = 0;
lkp->lk_prio = prio;
lkp->lk_wmesg = wmesg;
lkp->lk_timo = timo;
lkp->lk_lockholder = LK_NOPROC;
Remove a race condition / deadlock from snapshots. When converting from individual vnode locks to the snapshot lock, be sure to pass any waiting processes along to the new lock as well. This transfer is done by a new function in the lock manager, transferlockers(from_lock, to_lock); Thanks to Lamont Granquist <lamont@scriptkiddie.org> for his help in pounding on snapshots beyond all reason and finding this deadlock. Sponsored by: DARPA & NAI Labs.
2002-11-30 19:00:51 +00:00
lkp->lk_newlock = NULL;
Have lockinit() initialize the debugging fields of a lock when DEBUG_LOCKS is defined. Sponsored by: DARPA & NAI Labs.
2002-10-18 01:34:10 +00:00
#ifdef DEBUG_LOCKS
- Fix a problem that slipped through review; the stack member of the lockmgr structure should have the lk_ prefix. - Add stack_print(lkp->lk_stack) to the information printed with lockmgr_printinfo().
2005-08-03 04:59:07 +00:00
stack_zero(&lkp->lk_stack);
Have lockinit() initialize the debugging fields of a lock when DEBUG_LOCKS is defined. Sponsored by: DARPA & NAI Labs.
2002-10-18 01:34:10 +00:00
#endif
track lock class name in a way that doesn't break WITNESS
2006-11-13 05:41:46 +00:00
lock_profile_object_init(&lkp->lk_object, &lock_class_lockmgr, wmesg);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
}
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
/*
* Destroy a lock.
*/
void
lockdestroy(lkp)
struct lock *lkp;
{
Use the KTR_LOCK mask for logging events via KTR in lockmgr() rather than KTR_LOCKMGR. lockmgr locks are locks just like other locks.
2003-03-11 20:00:37 +00:00
CTR2(KTR_LOCK, "lockdestroy(): lkp == %p (lk_wmesg == \"%s\")",
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
lkp, lkp->lk_wmesg);
MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profile wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb
2006-11-11 03:18:07 +00:00
lock_profile_object_destroy(&lkp->lk_object);
Convert lockmgr locks from using simple locks to using mutexes. Add lockdestroy() and appropriate invocations, which corresponds to lockinit() and must be called to clean up after a lockmgr lock is no longer needed.
2000-10-04 01:29:17 +00:00
}
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
/*
* Determine the status of a lock.
*/
int
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
lockstatus(lkp, td)
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
struct lock *lkp;
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
struct thread *td;
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
{
int lock_type = 0;
In lockstatus(), don't lock and unlock the interlock when testing the sleep lock status while kdb_active, or we risk contending with the mutex on another CPU, resulting in a panic when using "show lockedvnods" while in DDB. MFC after: 3 days Reviewed by: jhb Reported by: kris
2005-09-27 21:02:59 +00:00
int interlocked;
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
In lockstatus(), don't lock and unlock the interlock when testing the sleep lock status while kdb_active, or we risk contending with the mutex on another CPU, resulting in a panic when using "show lockedvnods" while in DDB. MFC after: 3 days Reviewed by: jhb Reported by: kris
2005-09-27 21:02:59 +00:00
if (!kdb_active) {
interlocked = 1;
mtx_lock(lkp->lk_interlock);
} else
interlocked = 0;
Lock reporting and assertion changes. * lockstatus() and VOP_ISLOCKED() gets a new process argument and a new return value: LK_EXCLOTHER, when the lock is held exclusively by another process. * The ASSERT_VOP_(UN)LOCKED family is extended to use what this gives them * Extend the vnode_if.src format to allow more exact specification than locked/unlocked. This commit should not do any semantic changes unless you are using DEBUG_VFS_LOCKS. Discussed with: grog, mch, peter, phk Reviewed by: peter
1999-12-11 16:13:02 +00:00
if (lkp->lk_exclusivecount != 0) {
The lockmanager has to keep track of locks per thread, not per process. Submitted by: david Xu (davidxu@) Reviewed by: jhb@
2003-02-05 19:36:58 +00:00
if (td == NULL || lkp->lk_lockholder == td)
Lock reporting and assertion changes. * lockstatus() and VOP_ISLOCKED() gets a new process argument and a new return value: LK_EXCLOTHER, when the lock is held exclusively by another process. * The ASSERT_VOP_(UN)LOCKED family is extended to use what this gives them * Extend the vnode_if.src format to allow more exact specification than locked/unlocked. This commit should not do any semantic changes unless you are using DEBUG_VFS_LOCKS. Discussed with: grog, mch, peter, phk Reviewed by: peter
1999-12-11 16:13:02 +00:00
lock_type = LK_EXCLUSIVE;
else
lock_type = LK_EXCLOTHER;
} else if (lkp->lk_sharecount != 0)
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
lock_type = LK_SHARED;
In lockstatus(), don't lock and unlock the interlock when testing the sleep lock status while kdb_active, or we risk contending with the mutex on another CPU, resulting in a panic when using "show lockedvnods" while in DDB. MFC after: 3 days Reviewed by: jhb Reported by: kris
2005-09-27 21:02:59 +00:00
if (interlocked)
mtx_unlock(lkp->lk_interlock);
Change the M_NAMEI allocations to use the zone allocator. This change plus the previous changes to use the zone allocator decrease the useage of malloc by half. The Zone allocator will be upgradeable to be able to use per CPU-pools, and has more intelligent usage of SPLs. Additionally, it has reasonable stats gathering capabilities, while making most calls inline.
1997-09-21 04:24:27 +00:00
return (lock_type);
}
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
/*
* Determine the number of holders of a lock.
*/
int
lockcount(lkp)
struct lock *lkp;
{
int count;
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(lkp->lk_interlock);
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
count = lkp->lk_exclusivecount + lkp->lk_sharecount;
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(lkp->lk_interlock);
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
return (count);
}
If the buffer lock has waiters after the buffer has changed identity then getnewbuf() needs to drop the buffer in order to wake waiters that might sleep on the buffer in the context of the old identity.
2006-10-02 02:06:27 +00:00
/*
* Determine the number of waiters on a lock.
*/
int
lockwaiters(lkp)
struct lock *lkp;
{
int count;
mtx_lock(lkp->lk_interlock);
count = lkp->lk_waitcount;
mtx_unlock(lkp->lk_interlock);
return (count);
}
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
/*
* Print out information about state of a lock. Used by VOP_PRINT
typo in comment.
1997-08-22 07:16:46 +00:00
* routines to display status about contained locks.
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
*/
Replace original revision 1.1; Author dyson; Date: 1997/02/10 02:28:15 Changes from Lite2: - DEBUG -> SIMPLELOCK_DEBUG - cosmetic fixes - bzero of lock at init time -> explicit init of members.
1997-03-25 16:32:46 +00:00
void
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
lockmgr_printinfo(lkp)
struct lock *lkp;
{
if (lkp->lk_sharecount)
printf(" lock type %s: SHARED (count %d)", lkp->lk_wmesg,
lkp->lk_sharecount);
else if (lkp->lk_flags & LK_HAVE_EXCL)
Add pid to the info printed in lockmgr_printinfo. This makes VFS diagnostic messages slightly more useful.
2004-01-06 04:34:13 +00:00
printf(" lock type %s: EXCL (count %d) by thread %p (pid %d)",
lkp->lk_wmesg, lkp->lk_exclusivecount,
lkp->lk_lockholder, lkp->lk_lockholder->td_proc->p_pid);
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
if (lkp->lk_waitcount > 0)
printf(" with %d pending", lkp->lk_waitcount);
- Fix a problem that slipped through review; the stack member of the lockmgr structure should have the lk_ prefix. - Add stack_print(lkp->lk_stack) to the information printed with lockmgr_printinfo().
2005-08-03 04:59:07 +00:00
#ifdef DEBUG_LOCKS
stack_print(&lkp->lk_stack);
#endif
Import 4.4BSD-Lite2 onto CSRG branch
1997-03-25 16:27:20 +00:00
}
Add a 'show lockmgr' command that dumps the relevant details of a lockmgr lock.
2006-08-15 16:42:16 +00:00
#ifdef DDB
Add a new 'show sleepchain' ddb command similar to 'show lockchain' except that it operates on lockmgr and sx locks. This can be useful for tracking down vnode deadlocks in VFS for example. Note that this command is a bit more fragile than 'show lockchain' as we have to poke around at the wait channel of a thread to see if it points to either a struct lock or a condition variable inside of a struct sx. If td_wchan points to something unmapped, then this command will terminate early due to a fault, but no harm will be done.
2006-08-15 18:29:01 +00:00
/*
* Check to see if a thread that is blocked on a sleep queue is actually
* blocked on a 'struct lock'. If so, output some details and return true.
* If the lock has an exclusive owner, return that in *ownerp.
*/
int
lockmgr_chain(struct thread *td, struct thread **ownerp)
{
struct lock *lkp;
lkp = td->td_wchan;
/* Simple test to see if wchan points to a lockmgr lock. */
if (lkp->lk_wmesg != td->td_wmesg)
return (0);
/* Ok, we think we have a lockmgr lock, so output some details. */
db_printf("blocked on lk \"%s\" ", lkp->lk_wmesg);
if (lkp->lk_sharecount) {
db_printf("SHARED (count %d)\n", lkp->lk_sharecount);
*ownerp = NULL;
} else {
db_printf("EXCL (count %d)\n", lkp->lk_exclusivecount);
*ownerp = lkp->lk_lockholder;
}
return (1);
}
track lock class name in a way that doesn't break WITNESS
2006-11-13 05:41:46 +00:00
void
db_show_lockmgr(struct lock_object *lock)
Add a 'show lockmgr' command that dumps the relevant details of a lockmgr lock.
2006-08-15 16:42:16 +00:00
{
struct thread *td;
struct lock *lkp;
track lock class name in a way that doesn't break WITNESS
2006-11-13 05:41:46 +00:00
lkp = (struct lock *)lock;
Add a 'show lockmgr' command that dumps the relevant details of a lockmgr lock.
2006-08-15 16:42:16 +00:00
db_printf("lock type: %s\n", lkp->lk_wmesg);
db_printf("state: ");
if (lkp->lk_sharecount)
db_printf("SHARED (count %d)\n", lkp->lk_sharecount);
else if (lkp->lk_flags & LK_HAVE_EXCL) {
td = lkp->lk_lockholder;
db_printf("EXCL (count %d) %p ", lkp->lk_exclusivecount, td);
db_printf("(tid %d, pid %d, \"%s\")\n", td->td_tid,
td->td_proc->p_pid, td->td_proc->p_comm);
} else
db_printf("UNLOCKED\n");
if (lkp->lk_waitcount > 0)
db_printf("waiters: %d\n", lkp->lk_waitcount);
}
#endif
Reference in New Issue Copy Permalink