d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7a54260771
freebsd-dev/sys/kern/kern_malloc.c

566 lines
15 KiB
C
Raw Normal View History

BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Copyright (c) 1987, 1991, 1993
* The Regents of the University of California. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. All advertising materials mentioning features or use of this software
* must display the following acknowledgement:
* This product includes software developed by the University of
* California, Berkeley and its contributors.
* 4. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)kern_malloc.c 8.3 (Berkeley) 1/4/94
$Id$ -> $FreeBSD$
1999-08-28 01:08:13 +00:00
* $FreeBSD$
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#include "opt_vm.h"
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/param.h>
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
#include <sys/systm.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/kernel.h>
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files. Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files. Sort sys/*.h includes where possible in affected files. OK'ed by: bde (with reservations)
2001-05-01 08:13:21 +00:00
#include <sys/lock.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/malloc.h>
Fixed two bugs in the calculation of the malloc area (kmem_map) size: 1) The calculation didn't account for NMBCLUSTERS, so if a large number of clusters was specified, it would leave little or no space for kernel malloc. 2) It was bogusly restricted to v_page_count. This doesn't take into account the sparseness of the malloc area and would have caused problems on machines with small amounts of memory. It should probably instead be changed to set the malloc limit to be constrained by the amount of memory, but I didn't do this.
1996-01-29 09:58:34 +00:00
#include <sys/mbuf.h>
- machine/mutex.h -> sys/mutex.h - Use MUTEX_DECLARE() and MTX_COLD for the malloc_mtx mutex
2000-10-20 07:29:16 +00:00
#include <sys/mutex.h>
Untangled the vm.h include file spaghetti.
1995-12-07 12:48:31 +00:00
#include <sys/vmmeter.h>
Make intr_nesting_level per-process, rather than per-cpu. Setup interrupt threads to run with it always >= 1, so that malloc can detect M_WAITOK from "interrupt" context. This is also necessary in order to context switch from sched_ithd() directly. Reviewed By: peter
2001-01-21 19:25:07 +00:00
#include <sys/proc.h>
Don't #include <sys/proc.h>, since machine/mutex.h does it now.
2000-09-23 00:01:37 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <vm/vm.h>
Untangled the vm.h include file spaghetti.
1995-12-07 12:48:31 +00:00
#include <vm/vm_param.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <vm/vm_kern.h>
Untangled the vm.h include file spaghetti.
1995-12-07 12:48:31 +00:00
#include <vm/vm_extern.h>
Get rid of the ad-hoc memory allocator for vm_map_entries, in lieu of a simple, clean zone type allocator. This new allocator will also be used for machine dependent pmap PV entries.
1997-08-05 00:02:08 +00:00
#include <vm/pmap.h>
#include <vm/vm_map.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
KASSERT that we cannot use M_WAITOK in interrupt context. Reviewed by: bde
1999-09-19 08:40:11 +00:00
#if defined(INVARIANTS) && defined(__i386__)
#include <machine/cpu.h>
#endif
Before we start to mess with the VFS name-cache clean things up a little bit: Isolate the namecache in its own file, and give it a dedicated malloc type.
1999-10-03 12:18:29 +00:00
MALLOC_DEFINE(M_CACHE, "cache", "Various Dynamically allocated caches");
Get rid of MALLOC_INSTANTIATE and MALLOC_MAKE_TYPE(). Just handle the 3 malloc types declared in <sys/malloc.h> like other global malloc types.
1999-09-11 16:41:39 +00:00
MALLOC_DEFINE(M_DEVBUF, "devbuf", "device driver memory");
MALLOC_DEFINE(M_TEMP, "temp", "misc temporary data buffers");
KAME netinet6 basic part(no IPsec,no V6 Multicast Forwarding, no UDP/TCP for IPv6 yet) With this patch, you can assigne IPv6 addr automatically, and can reply to IPv6 ping. Reviewed by: freebsd-arch, cvs-committers Obtained from: KAME project
1999-11-22 02:45:11 +00:00
MALLOC_DEFINE(M_IP6OPT, "ip6opt", "IPv6 options");
MALLOC_DEFINE(M_IP6NDP, "ip6ndp", "IPv6 Neighbor Discovery");
Fixed init functions argument type - caddr_t -> void *. Fixed a couple of compiler warnings.
1995-09-09 18:10:37 +00:00
static void kmeminit __P((void *));
Reviewed by: julian with quick glances by bruce and others Submitted by: terry (terry lambert) This is a composite of 3 patch sets submitted by terry. they are: New low-level init code that supports loadbal modules better some cleanups in the namei code to help terry in 16-bit character support some changes to the mount-root code to make it a little more modular.. NOTE: mounting root off cdrom or NFS MIGHT be broken as I haven't been able to test those cases.. certainly mounting root of disk still works just fine.. mfs should work but is untested. (tomorrows task) The low level init stuff includes a total rewrite of init_main.c to make it possible for new modules to have an init phase by simply adding an entry to a TEXT_SET (or is it DATA_SET) list. thus a new module can be added to the kernel without editing any other files other than the 'files' file.
1995-08-28 09:19:25 +00:00
SYSINIT(kmem, SI_SUB_KMEM, SI_ORDER_FIRST, kmeminit, NULL)
Last major round (Unless Bruce thinks of somthing :-) of malloc changes. Distribute all but the most fundamental malloc types. This time I also remembered the trick to making things static: Put "static" in front of them. A couple of finer points by: bde
1997-10-12 20:26:33 +00:00
static MALLOC_DEFINE(M_FREE, "free", "should be on free list");
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
static struct malloc_type *kmemstatistics;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
static struct kmembuckets bucket[MINBUCKET + 16];
static struct kmemusage *kmemusage;
static char *kmembase;
Fixed staticization. buckets[] was staticized but was still declared extern in <sys/malloc.h> and it should not have been staticized for the !(KMEMSTATS || DIAGNOSTIC) case. Fixed the !(KMEMSTATS || DIAGNOSTIC) case. The MALLOC() and FREE() macros are evil, but code generally doesn't allow for this and some code involving else clauses did not compile. Finished staticization.
1997-09-16 13:52:04 +00:00
static char *kmemlimit;
Fix null-pointer dereference crash when the system is intentionally run out of KVM through a mmap()/fork() bomb that allocates hundreds of thousands of vm_map_entry structures. Add panic to make null-pointer dereference crash a little more verbose. Add a new sysctl, vm.max_proc_mmap, which specifies the maximum number of mmap()'d spaces (discrete vm_map_entry's in the process). The value defaults to around 9000 for a 128MB machine. The test is scaled for the number of processes sharing a vmspace (aka linux threads). Setting the value to 0 disables the feature. PR: kern/16573 Approved by: jkh
2000-02-16 21:11:33 +00:00
Remove MUTEX_DECLARE() and MTX_COLD. Instead, postpone full mutex initialization until after malloc() is safe to call, then iterate through all mutexes and complete their initialization. This change is necessary in order to avoid some circular bootstrapping dependencies.
2001-01-21 07:52:20 +00:00
static struct mtx malloc_mtx;
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
Fix null-pointer dereference crash when the system is intentionally run out of KVM through a mmap()/fork() bomb that allocates hundreds of thousands of vm_map_entry structures. Add panic to make null-pointer dereference crash a little more verbose. Add a new sysctl, vm.max_proc_mmap, which specifies the maximum number of mmap()'d spaces (discrete vm_map_entry's in the process). The value defaults to around 9000 for a 128MB machine. The test is scaled for the number of processes sharing a vmspace (aka linux threads). Setting the value to 0 disables the feature. PR: kern/16573 Approved by: jkh
2000-02-16 21:11:33 +00:00
u_int vm_kmem_size;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
KNFize, by bde.
1999-01-10 01:58:29 +00:00
#ifdef INVARIANTS
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* This structure provides a set of masks to catch unaligned frees.
*/
A Major staticize sweep. Generates a couple of warnings that I'll deal with later. A number of unused vars removed. A number of unused procs removed or #ifdefed.
1995-12-14 08:32:45 +00:00
static long addrmask[] = { 0,
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
0x00000001, 0x00000003, 0x00000007, 0x0000000f,
0x0000001f, 0x0000003f, 0x0000007f, 0x000000ff,
0x000001ff, 0x000003ff, 0x000007ff, 0x00000fff,
0x00001fff, 0x00003fff, 0x00007fff, 0x0000ffff,
};
/*
* The WEIRD_ADDR is used as known text to copy into free objects so
* that modifications after frees can be detected.
*/
Make vegetarian and animal rights people happy and use 0xdeadc0de instead of 0xdeadbeef as the fill pattern. Decreased MAX_COPY to 64 (256 was a bit overzealous in most cases).
1995-04-16 11:25:15 +00:00
#define WEIRD_ADDR 0xdeadc0de
#define MAX_COPY 64
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Normally the first word of the structure is used to hold the list
* pointer for free objects. However, when running with diagnostics,
* we use the third and fourth fields, so as to catch modifications
* in the most commonly trashed first two words.
*/
struct freelist {
long spare0;
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
struct malloc_type *type;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
long spare1;
caddr_t next;
};
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#else /* !INVARIANTS */
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
struct freelist {
caddr_t next;
};
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#endif /* INVARIANTS */
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
This is a rather large commit that encompasses the new swapper, changes to the VM system to support the new swapper, VM bug fixes, several VM optimizations, and some additional revamping of the VM code. The specific bug fixes will be documented with additional forced commits. This commit is somewhat rough in regards to code cleanup issues. Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>
1999-01-21 08:29:12 +00:00
* malloc:
*
* Allocate a block of memory.
*
* If M_NOWAIT is set, this routine will not block and return NULL if
* the allocation fails.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
void *
malloc(size, type, flags)
unsigned long size;
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
struct malloc_type *type;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
int flags;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
register struct kmembuckets *kbp;
register struct kmemusage *kup;
register struct freelist *freep;
long indx, npg, allocsize;
int s;
caddr_t va, cp, savedlist;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#ifdef INVARIANTS
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
long *end, *lp;
int copysize;
Fix warnings in preparation for adding -Wall -Wcast-qual to the kernel compile
1999-01-27 21:50:00 +00:00
const char *savedtype;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#endif
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
register struct malloc_type *ksp = type;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
p->p_intr_nesting_level is MI now and initialized to 0 in kern_fork.c, so it should be save to KASSERT() on it even on an arch that may not use it.
2001-01-27 06:32:20 +00:00
#if defined(INVARIANTS)
KASSERT that we cannot use M_WAITOK in interrupt context. Reviewed by: bde
1999-09-19 08:40:11 +00:00
if (flags == M_WAITOK)
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
KASSERT(curthread->td_intr_nesting_level == 0,
KASSERT that we cannot use M_WAITOK in interrupt context. Reviewed by: bde
1999-09-19 08:40:11 +00:00
("malloc(M_WAITOK) in interrupt context"));
#endif
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
indx = BUCKETINDX(size);
kbp = &bucket[indx];
sys/malloc.h: Order the SYSINIT() for MALLOC_DEFINE() correctly so that malloc() doesn't have to waste time initializing itself. The (SI_SUB_KMEM, SI_ORDER_ANY) order was shared with syscons' SYSINIT() for scmeminit(), and scmeminit() calls malloc(), so malloc() initialization was not always complete on the first call to malloc(). kern/kern_malloc.c: - Removed self-initialization in malloc(). - Removed half-baked sanity check in free(). Trust MALLOC_DEFINE().
2000-06-14 18:31:42 +00:00
s = splmem();
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(&malloc_mtx);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
while (ksp->ks_memuse >= ksp->ks_limit) {
if (flags & M_NOWAIT) {
splx(s);
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&malloc_mtx);
Let M_PANIC go back to the private tree as its intention isn't understood well for now.
2001-01-31 04:50:20 +00:00
return ((void *) NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
if (ksp->ks_limblocks < 65535)
ksp->ks_limblocks++;
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
msleep((caddr_t)ksp, &malloc_mtx, PSWP+2, type->ks_shortdesc,
0);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
ksp->ks_size |= 1 << indx;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#ifdef INVARIANTS
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
copysize = 1 << indx < MAX_COPY ? 1 << indx : MAX_COPY;
#endif
if (kbp->kb_next == NULL) {
kbp->kb_last = NULL;
if (size > MAXALLOCSAVE)
First pass at cleaning up macros relating to pages, clusters and all that.
1996-05-02 10:43:17 +00:00
allocsize = roundup(size, PAGE_SIZE);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
else
allocsize = 1 << indx;
removed: CLBYTES PD_SHIFT PGSHIFT NBPG PGOFSET CLSIZELOG2 CLSIZE pdei() ptei() kvtopte() ptetov() ispt() ptetoav() &c &c new: NPDEPG Major macro cleanup.
1996-05-02 14:21:14 +00:00
npg = btoc(allocsize);
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&malloc_mtx);
Calling semantics for kmem_malloc() have been changed...and the third argument is now more than just a single flag. (kern_malloc.c) Used new M_KERNEL value for socket allocations that previous were "M_NOWAIT". Note that this will change when we clean up the M_ namespace mess. Submitted by: John Dyson
1995-02-02 08:49:08 +00:00
va = (caddr_t) kmem_malloc(kmem_map, (vm_size_t)ctob(npg), flags);
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (va == NULL) {
splx(s);
Let M_PANIC go back to the private tree as its intention isn't understood well for now.
2001-01-31 04:50:20 +00:00
return ((void *) NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
/*
* Enter malloc_mtx after the error check to avoid having to
* immediately exit it again if there is an error.
*/
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(&malloc_mtx);
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kbp->kb_total += kbp->kb_elmpercl;
kup = btokup(va);
kup->ku_indx = indx;
if (allocsize > MAXALLOCSAVE) {
if (npg > 65535)
panic("malloc: allocation too large");
kup->ku_pagecnt = npg;
ksp->ks_memuse += allocsize;
goto out;
}
kup->ku_freecnt = kbp->kb_elmpercl;
kbp->kb_totalfree += kbp->kb_elmpercl;
/*
* Just in case we blocked while allocating memory,
* and someone else also allocated memory for this
* bucket, don't assume the list is still empty.
*/
savedlist = kbp->kb_next;
removed: CLBYTES PD_SHIFT PGSHIFT NBPG PGOFSET CLSIZELOG2 CLSIZE pdei() ptei() kvtopte() ptetov() ispt() ptetoav() &c &c new: NPDEPG Major macro cleanup.
1996-05-02 14:21:14 +00:00
kbp->kb_next = cp = va + (npg * PAGE_SIZE) - allocsize;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
for (;;) {
freep = (struct freelist *)cp;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#ifdef INVARIANTS
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Copy in known text to detect modification
* after freeing.
*/
end = (long *)&cp[copysize];
for (lp = (long *)cp; lp < end; lp++)
*lp = WEIRD_ADDR;
freep->type = M_FREE;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#endif /* INVARIANTS */
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (cp <= va)
break;
cp -= allocsize;
freep->next = cp;
}
freep->next = savedlist;
if (kbp->kb_last == NULL)
kbp->kb_last = (caddr_t)freep;
}
va = kbp->kb_next;
kbp->kb_next = ((struct freelist *)va)->next;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#ifdef INVARIANTS
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
freep = (struct freelist *)va;
Print "previous type" correctly when INVARIANTS is defined. Reviewed by: current@FreeBSD.org
2000-03-14 14:58:04 +00:00
savedtype = (const char *) freep->type->ks_shortdesc;
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
freep->type = (struct malloc_type *)WEIRD_ADDR;
Use [u]intptr_t instead of [u_]long for casts between pointers and integers. Don't forget to cast to (void *) as well.
1998-08-16 01:21:52 +00:00
if ((intptr_t)(void *)&freep->next & 0x2)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
freep->next = (caddr_t)((WEIRD_ADDR >> 16)|(WEIRD_ADDR << 16));
else
freep->next = (caddr_t)WEIRD_ADDR;
end = (long *)&va[copysize];
for (lp = (long *)va; lp < end; lp++) {
if (*lp == WEIRD_ADDR)
continue;
Fixed printf format errors.
1998-07-29 17:38:14 +00:00
printf("%s %ld of object %p size %lu %s %s (0x%lx != 0x%lx)\n",
"Data modified on freelist: word",
(long)(lp - (long *)va), (void *)va, size,
"previous type", savedtype, *lp, (u_long)WEIRD_ADDR);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
break;
}
freep->spare0 = 0;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#endif /* INVARIANTS */
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kup = btokup(va);
if (kup->ku_indx != indx)
panic("malloc: wrong bucket");
if (kup->ku_freecnt == 0)
panic("malloc: lost data");
kup->ku_freecnt--;
kbp->kb_totalfree--;
ksp->ks_memuse += 1 << indx;
out:
kbp->kb_calls++;
ksp->ks_inuse++;
ksp->ks_calls++;
if (ksp->ks_memuse > ksp->ks_maxused)
ksp->ks_maxused = ksp->ks_memuse;
splx(s);
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&malloc_mtx);
Introduce the M_ZERO flag to malloc(9) Instead of: foo = malloc(sizeof(foo), M_WAIT); bzero(foo, sizeof(foo)); You can now (and please do) use: foo = malloc(sizeof(foo), M_WAIT | M_ZERO); In the future this will enable us to do idle-time pre-zeroing of malloc-space.
2000-10-20 17:54:55 +00:00
/* XXX: Do idle pre-zeroing. */
if (va != NULL && (flags & M_ZERO))
bzero(va, size);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
return ((void *) va);
}
/*
This is a rather large commit that encompasses the new swapper, changes to the VM system to support the new swapper, VM bug fixes, several VM optimizations, and some additional revamping of the VM code. The specific bug fixes will be documented with additional forced commits. This commit is somewhat rough in regards to code cleanup issues. Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>
1999-01-21 08:29:12 +00:00
* free:
*
* Free a block of memory allocated by malloc.
*
* This routine may not block.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
void
free(addr, type)
void *addr;
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
struct malloc_type *type;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
register struct kmembuckets *kbp;
register struct kmemusage *kup;
register struct freelist *freep;
long size;
int s;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#ifdef INVARIANTS
The check for multiple freed items were bogus. fixed.
1996-08-04 20:08:48 +00:00
struct freelist *fp;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
long *end, *lp, alloc, copysize;
#endif
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
register struct malloc_type *ksp = type;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
KASSERT(kmembase <= (char *)addr && (char *)addr < kmemlimit,
KNFize, by bde.
1999-01-10 01:58:29 +00:00
("free: address %p out of range", (void *)addr));
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kup = btokup(addr);
size = 1 << kup->ku_indx;
kbp = &bucket[kup->ku_indx];
Reviewed by: dyson@freebsd.org (john Dyson), dg@root.com (david greenman) Submitted by: Kirk McKusick (mcKusick@mckusick.com) Obtained from: WHistle development tree
1998-03-08 09:59:44 +00:00
s = splmem();
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(&malloc_mtx);
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#ifdef INVARIANTS
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Check for returns of data that do not point to the
* beginning of the allocation.
*/
First pass at cleaning up macros relating to pages, clusters and all that.
1996-05-02 10:43:17 +00:00
if (size > PAGE_SIZE)
alloc = addrmask[BUCKETINDX(PAGE_SIZE)];
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
else
alloc = addrmask[kup->ku_indx];
Use [u]intptr_t instead of [u_]long for casts between pointers and integers. Don't forget to cast to (void *) as well.
1998-08-16 01:21:52 +00:00
if (((uintptr_t)(void *)addr & alloc) != 0)
Fixed printf format errors.
1998-07-29 17:38:14 +00:00
panic("free: unaligned addr %p, size %ld, type %s, mask %ld",
(void *)addr, size, type->ks_shortdesc, alloc);
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#endif /* INVARIANTS */
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (size > MAXALLOCSAVE) {
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&malloc_mtx);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kmem_free(kmem_map, (vm_offset_t)addr, ctob(kup->ku_pagecnt));
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(&malloc_mtx);
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
removed: CLBYTES PD_SHIFT PGSHIFT NBPG PGOFSET CLSIZELOG2 CLSIZE pdei() ptei() kvtopte() ptetov() ispt() ptetoav() &c &c new: NPDEPG Major macro cleanup.
1996-05-02 14:21:14 +00:00
size = kup->ku_pagecnt << PAGE_SHIFT;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
ksp->ks_memuse -= size;
kup->ku_indx = 0;
kup->ku_pagecnt = 0;
if (ksp->ks_memuse + size >= ksp->ks_limit &&
ksp->ks_memuse < ksp->ks_limit)
wakeup((caddr_t)ksp);
ksp->ks_inuse--;
kbp->kb_total -= 1;
splx(s);
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&malloc_mtx);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
return;
}
freep = (struct freelist *)addr;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#ifdef INVARIANTS
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Check for multiple frees. Use a quick check to see if
* it looks free before laboriously searching the freelist.
*/
if (freep->spare0 == WEIRD_ADDR) {
The check for multiple freed items were bogus. fixed.
1996-08-04 20:08:48 +00:00
fp = (struct freelist *)kbp->kb_next;
while (fp) {
KNFize, by bde.
1999-01-10 01:58:29 +00:00
if (fp->spare0 != WEIRD_ADDR)
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
panic("free: free item %p modified", fp);
KNFize, by bde.
1999-01-10 01:58:29 +00:00
else if (addr == (caddr_t)fp)
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
panic("free: multiple freed item %p", addr);
The check for multiple freed items were bogus. fixed.
1996-08-04 20:08:48 +00:00
fp = (struct freelist *)fp->next;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
}
/*
* Copy in known text to detect modification after freeing
* and to make it look free. Also, save the type being freed
* so we can list likely culprit if modification is detected
* when the object is reallocated.
*/
copysize = size < MAX_COPY ? size : MAX_COPY;
end = (long *)&((caddr_t)addr)[copysize];
for (lp = (long *)addr; lp < end; lp++)
*lp = WEIRD_ADDR;
freep->type = type;
Split DIAGNOSTIC -> DIAGNOSTIC, INVARIANTS, and INVARIANT_SUPPORT as discussed on -hackers. Introduce 'KASSERT(assertion, ("panic message", args))' for simple check + panic. Reviewed by: msmith
1999-01-08 17:31:30 +00:00
#endif /* INVARIANTS */
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kup->ku_freecnt++;
Add sufficient braces to keep egcs happy about potentially ambiguous if/else nesting.
1999-05-06 18:13:11 +00:00
if (kup->ku_freecnt >= kbp->kb_elmpercl) {
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (kup->ku_freecnt > kbp->kb_elmpercl)
panic("free: multiple frees");
else if (kbp->kb_totalfree > kbp->kb_highwat)
kbp->kb_couldfree++;
Add sufficient braces to keep egcs happy about potentially ambiguous if/else nesting.
1999-05-06 18:13:11 +00:00
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kbp->kb_totalfree++;
ksp->ks_memuse -= size;
if (ksp->ks_memuse + size >= ksp->ks_limit &&
ksp->ks_memuse < ksp->ks_limit)
wakeup((caddr_t)ksp);
ksp->ks_inuse--;
Minor performance improvement to kern_malloc.c that increases the probability of reuse of recently freed memory. This improves cache hit stats on cached memory, and improves at least fork speed consistancy.
1996-05-18 22:33:13 +00:00
#ifdef OLD_MALLOC_MEMORY_POLICY
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (kbp->kb_next == NULL)
kbp->kb_next = addr;
else
((struct freelist *)kbp->kb_last)->next = addr;
freep->next = NULL;
kbp->kb_last = addr;
Minor performance improvement to kern_malloc.c that increases the probability of reuse of recently freed memory. This improves cache hit stats on cached memory, and improves at least fork speed consistancy.
1996-05-18 22:33:13 +00:00
#else
/*
* Return memory to the head of the queue for quick reuse. This
* can improve performance by improving the probability of the
* item being in the cache when it is reused.
*/
if (kbp->kb_next == NULL) {
kbp->kb_next = addr;
kbp->kb_last = addr;
freep->next = NULL;
} else {
freep->next = kbp->kb_next;
kbp->kb_next = addr;
}
#endif
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
splx(s);
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&malloc_mtx);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
/*
* Initialize the kernel memory allocator
*/
Reviewed by: julian with quick glances by bruce and others Submitted by: terry (terry lambert) This is a composite of 3 patch sets submitted by terry. they are: New low-level init code that supports loadbal modules better some cleanups in the namei code to help terry in 16-bit character support some changes to the mount-root code to make it a little more modular.. NOTE: mounting root off cdrom or NFS MIGHT be broken as I haven't been able to test those cases.. certainly mounting root of disk still works just fine.. mfs should work but is untested. (tomorrows task) The low level init stuff includes a total rewrite of init_main.c to make it possible for new modules to have an init phase by simply adding an entry to a TEXT_SET (or is it DATA_SET) list. thus a new module can be added to the kernel without editing any other files other than the 'files' file.
1995-08-28 09:19:25 +00:00
/* ARGSUSED*/
static void
Finished (?) cleaning up sysinit stuff.
1995-12-02 17:11:20 +00:00
kmeminit(dummy)
void *dummy;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
register long indx;
Fixed sign and overflow bugs that caused the allocation size of the kernel malloc region (kmem_map) to be wrong and semi-random on systems with more than 1GB of RAM. This is not a complete fix, but is sufficient for machines with 4GB or less of memory. A complete fix will require some changes to the getenv stuff so that 64bit values can be passed around. NOT FIXED: machines with more than 4GB of RAM (e.g. some large Alphas) since we're still using ints to hold some of the values. Reviewed by: bde
2000-01-28 04:04:58 +00:00
u_long npg;
u_long mem_size;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#if ((MAXALLOCSAVE & (MAXALLOCSAVE - 1)) != 0)
Allocate mbufs from a separate submap so that NMBCLUSTERS works as expected.
1996-05-10 19:28:55 +00:00
#error "kmeminit: MAXALLOCSAVE not power of 2"
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#endif
#if (MAXALLOCSAVE > MINALLOCSIZE * 32768)
Allocate mbufs from a separate submap so that NMBCLUSTERS works as expected.
1996-05-10 19:28:55 +00:00
#error "kmeminit: MAXALLOCSAVE too big"
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#endif
First pass at cleaning up macros relating to pages, clusters and all that.
1996-05-02 10:43:17 +00:00
#if (MAXALLOCSAVE < PAGE_SIZE)
Allocate mbufs from a separate submap so that NMBCLUSTERS works as expected.
1996-05-10 19:28:55 +00:00
#error "kmeminit: MAXALLOCSAVE too small"
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#endif
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
Remove MUTEX_DECLARE() and MTX_COLD. Instead, postpone full mutex initialization until after malloc() is safe to call, then iterate through all mutexes and complete their initialization. This change is necessary in order to avoid some circular bootstrapping dependencies.
2001-01-21 07:52:20 +00:00
mtx_init(&malloc_mtx, "malloc", MTX_DEF);
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
/*
* Try to auto-tune the kernel memory size, so that it is
* more applicable for a wider range of machine sizes.
* On an X86, a VM_KMEM_SIZE_SCALE value of 4 is good, while
* a VM_KMEM_SIZE of 12MB is a fair compromise. The
* VM_KMEM_SIZE_MAX is dependent on the maximum KVA space
* available, and on an X86 with a total KVA space of 256MB,
* try to keep VM_KMEM_SIZE_MAX at 80MB or below.
*
* Note that the kmem_map is also used by the zone allocator,
* so make sure that there is enough space.
*/
Make the TUNABLE_*() macros look and behave more consistantly like the SYSCTL_*() macros. TUNABLE_INT_DECL() was an odd name because it didn't actually declare the int, which is what the name suggests it would do.
2001-06-06 22:17:08 +00:00
vm_kmem_size = VM_KMEM_SIZE;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
mem_size = cnt.v_page_count * PAGE_SIZE;
#if defined(VM_KMEM_SIZE_SCALE)
Make the TUNABLE_*() macros look and behave more consistantly like the SYSCTL_*() macros. TUNABLE_INT_DECL() was an odd name because it didn't actually declare the int, which is what the name suggests it would do.
2001-06-06 22:17:08 +00:00
if ((mem_size / VM_KMEM_SIZE_SCALE) > vm_kmem_size)
vm_kmem_size = mem_size / VM_KMEM_SIZE_SCALE;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#endif
#if defined(VM_KMEM_SIZE_MAX)
Make the TUNABLE_*() macros look and behave more consistantly like the SYSCTL_*() macros. TUNABLE_INT_DECL() was an odd name because it didn't actually declare the int, which is what the name suggests it would do.
2001-06-06 22:17:08 +00:00
if (vm_kmem_size >= VM_KMEM_SIZE_MAX)
vm_kmem_size = VM_KMEM_SIZE_MAX;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#endif
Allow VM_KMEM_SIZE to be tuned from the kernel environment. This tuning value *completely* overrides any value precalculated by the kernel.
1999-01-21 21:54:32 +00:00
/* Allow final override from the kernel environment */
"Fix" the previous initial attempt at fixing TUNABLE_INT(). This time around, use a common function for looking up and extracting the tunables from the kernel environment. This saves duplicating the same function over and over again. This way typically has an overhead of 8 bytes + the path string, versus about 26 bytes + the path string.
2001-06-08 05:24:21 +00:00
TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size);
Allow VM_KMEM_SIZE to be tuned from the kernel environment. This tuning value *completely* overrides any value precalculated by the kernel.
1999-01-21 21:54:32 +00:00
Fixed sign and overflow bugs that caused the allocation size of the kernel malloc region (kmem_map) to be wrong and semi-random on systems with more than 1GB of RAM. This is not a complete fix, but is sufficient for machines with 4GB or less of memory. A complete fix will require some changes to the getenv stuff so that 64bit values can be passed around. NOT FIXED: machines with more than 4GB of RAM (e.g. some large Alphas) since we're still using ints to hold some of the values. Reviewed by: bde
2000-01-28 04:04:58 +00:00
/*
* Limit kmem virtual size to twice the physical memory.
* This allows for kmem map sparseness, but limits the size
* to something sane. Be careful to not overflow the 32bit
* ints while doing the check.
*/
if ((vm_kmem_size / 2) > (cnt.v_page_count * PAGE_SIZE))
vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
Introduce numerous SMP friendly changes to the mbuf allocator. Namely, introduce a modified allocation mechanism for mbufs and mbuf clusters; one which can scale under SMP and which offers the possibility of resource reclamation to be implemented in the future. Notable advantages: o Reduce contention for SMP by offering per-CPU pools and locks. o Better use of data cache due to per-CPU pools. o Much less code cache pollution due to excessively large allocation macros. o Framework for `grouping' objects from same page together so as to be able to possibly free wired-down pages back to the system if they are no longer needed by the network stacks. Additional things changed with this addition: - Moved some mbuf specific declarations and initializations from sys/conf/param.c into mbuf-specific code where they belong. - m_getclr() has been renamed to m_get_clrd() because the old name is really confusing. m_getclr() HAS been preserved though and is defined to the new name. No tree sweep has been done "to change the interface," as the old name will continue to be supported and is not depracated. The change was merely done because m_getclr() sounds too much like "m_get a cluster." - TEMPORARILY disabled mbtypes statistics displaying in netstat(1) and systat(1) (see TODO below). - Fixed systat(1) to display number of "free mbufs" based on new per-CPU stat structures. - Fixed netstat(1) to display new per-CPU stats based on sysctl-exported per-CPU stat structures. All infos are fetched via sysctl. TODO (in order of priority): - Re-enable mbtypes statistics in both netstat(1) and systat(1) after introducing an SMP friendly way to collect the mbtypes stats under the already introduced per-CPU locks (i.e. hopefully don't use atomic() - it seems too costly for a mere stat update, especially when other locks are already present). - Optionally have systat(1) display not only "total free mbufs" but also "total free mbufs per CPU pool." - Fix minor length-fetching issues in netstat(1) related to recently re-enabled option to read mbuf stats from a core file. - Move reference counters at least for mbuf clusters into an unused portion of the cluster itself, to save space and need to allocate a counter. - Look into introducing resource freeing possibly from a kproc. Reviewed by (in parts): jlemon, jake, silby, terry Tested by: jlemon (Intel & Alpha), mjacob (Intel & Alpha) Preliminary performance measurements: jlemon (and me, obviously) URL: http://people.freebsd.org/~bmilekic/mb_alloc/
2001-06-22 06:35:32 +00:00
/*
Rename mb_init() mbuf subsystem initialization routine to mbuf_init(), in order to avoid namespace collision with subr_mchain.c's mb_init(). This wasn't "fatal" as the mbuf initialization routine mb_init() was local to subr_mbuf.c which in turn didn't pull in subr_mchain.c's mb_init() declaration, but it should deffinately be changed now before it creates headache.
2001-08-03 05:05:32 +00:00
* In mbuf_init(), we set up submaps for mbufs and clusters, in which
Introduce numerous SMP friendly changes to the mbuf allocator. Namely, introduce a modified allocation mechanism for mbufs and mbuf clusters; one which can scale under SMP and which offers the possibility of resource reclamation to be implemented in the future. Notable advantages: o Reduce contention for SMP by offering per-CPU pools and locks. o Better use of data cache due to per-CPU pools. o Much less code cache pollution due to excessively large allocation macros. o Framework for `grouping' objects from same page together so as to be able to possibly free wired-down pages back to the system if they are no longer needed by the network stacks. Additional things changed with this addition: - Moved some mbuf specific declarations and initializations from sys/conf/param.c into mbuf-specific code where they belong. - m_getclr() has been renamed to m_get_clrd() because the old name is really confusing. m_getclr() HAS been preserved though and is defined to the new name. No tree sweep has been done "to change the interface," as the old name will continue to be supported and is not depracated. The change was merely done because m_getclr() sounds too much like "m_get a cluster." - TEMPORARILY disabled mbtypes statistics displaying in netstat(1) and systat(1) (see TODO below). - Fixed systat(1) to display number of "free mbufs" based on new per-CPU stat structures. - Fixed netstat(1) to display new per-CPU stats based on sysctl-exported per-CPU stat structures. All infos are fetched via sysctl. TODO (in order of priority): - Re-enable mbtypes statistics in both netstat(1) and systat(1) after introducing an SMP friendly way to collect the mbtypes stats under the already introduced per-CPU locks (i.e. hopefully don't use atomic() - it seems too costly for a mere stat update, especially when other locks are already present). - Optionally have systat(1) display not only "total free mbufs" but also "total free mbufs per CPU pool." - Fix minor length-fetching issues in netstat(1) related to recently re-enabled option to read mbuf stats from a core file. - Move reference counters at least for mbuf clusters into an unused portion of the cluster itself, to save space and need to allocate a counter. - Look into introducing resource freeing possibly from a kproc. Reviewed by (in parts): jlemon, jake, silby, terry Tested by: jlemon (Intel & Alpha), mjacob (Intel & Alpha) Preliminary performance measurements: jlemon (and me, obviously) URL: http://people.freebsd.org/~bmilekic/mb_alloc/
2001-06-22 06:35:32 +00:00
* case we rounddown() (nmbufs * MSIZE) and (nmbclusters * MCLBYTES),
* respectively. Mathematically, this means that what we do here may
* amount to slightly more address space than we need for the submaps,
* but it never hurts to have an extra page in kmem_map.
*/
Fix inconsistency in setup of kernel_map: we need to make sure that we also reserve _adequate_ space for the mb_map submap; i.e. we need space for nmbclusters, nmbufs, _and_ nmbcnt. Furthermore, we need to rounddown, and not roundup, so that we are consistent. Pointed out by: bde
2001-04-18 23:54:13 +00:00
npg = (nmbufs * MSIZE + nmbclusters * MCLBYTES + nmbcnt *
Introduce numerous SMP friendly changes to the mbuf allocator. Namely, introduce a modified allocation mechanism for mbufs and mbuf clusters; one which can scale under SMP and which offers the possibility of resource reclamation to be implemented in the future. Notable advantages: o Reduce contention for SMP by offering per-CPU pools and locks. o Better use of data cache due to per-CPU pools. o Much less code cache pollution due to excessively large allocation macros. o Framework for `grouping' objects from same page together so as to be able to possibly free wired-down pages back to the system if they are no longer needed by the network stacks. Additional things changed with this addition: - Moved some mbuf specific declarations and initializations from sys/conf/param.c into mbuf-specific code where they belong. - m_getclr() has been renamed to m_get_clrd() because the old name is really confusing. m_getclr() HAS been preserved though and is defined to the new name. No tree sweep has been done "to change the interface," as the old name will continue to be supported and is not depracated. The change was merely done because m_getclr() sounds too much like "m_get a cluster." - TEMPORARILY disabled mbtypes statistics displaying in netstat(1) and systat(1) (see TODO below). - Fixed systat(1) to display number of "free mbufs" based on new per-CPU stat structures. - Fixed netstat(1) to display new per-CPU stats based on sysctl-exported per-CPU stat structures. All infos are fetched via sysctl. TODO (in order of priority): - Re-enable mbtypes statistics in both netstat(1) and systat(1) after introducing an SMP friendly way to collect the mbtypes stats under the already introduced per-CPU locks (i.e. hopefully don't use atomic() - it seems too costly for a mere stat update, especially when other locks are already present). - Optionally have systat(1) display not only "total free mbufs" but also "total free mbufs per CPU pool." - Fix minor length-fetching issues in netstat(1) related to recently re-enabled option to read mbuf stats from a core file. - Move reference counters at least for mbuf clusters into an unused portion of the cluster itself, to save space and need to allocate a counter. - Look into introducing resource freeing possibly from a kproc. Reviewed by (in parts): jlemon, jake, silby, terry Tested by: jlemon (Intel & Alpha), mjacob (Intel & Alpha) Preliminary performance measurements: jlemon (and me, obviously) URL: http://people.freebsd.org/~bmilekic/mb_alloc/
2001-06-22 06:35:32 +00:00
sizeof(u_int) + vm_kmem_size) / PAGE_SIZE;
Remove trailing whitespace.
1995-05-30 08:16:23 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kmemusage = (struct kmemusage *) kmem_alloc(kernel_map,
(vm_size_t)(npg * sizeof(struct kmemusage)));
kmem_map = kmem_suballoc(kernel_map, (vm_offset_t *)&kmembase,
VM level code cleanups. 1) Start using TSM. Struct procs continue to point to upages structure, after being freed. Struct vmspace continues to point to pte object and kva space for kstack. u_map is now superfluous. 2) vm_map's don't need to be reference counted. They always exist either in the kernel or in a vmspace. The vmspaces are managed by reference counts. 3) Remove the "wired" vm_map nonsense. 4) No need to keep a cache of kernel stack kva's. 5) Get rid of strange looking ++var, and change to var++. 6) Change more data structures to use our "zone" allocator. Added struct proc, struct vmspace and struct vnode. This saves a significant amount of kva space and physical memory. Additionally, this enables TSM for the zone managed memory. 7) Keep ioopt disabled for now. 8) Remove the now bogus "single use" map concept. 9) Use generation counts or id's for data structures residing in TSM, where it allows us to avoid unneeded restart overhead during traversals, where blocking might occur. 10) Account better for memory deficits, so the pageout daemon will be able to make enough memory available (experimental.) 11) Fix some vnode locking problems. (From Tor, I think.) 12) Add a check in ufs_lookup, to avoid lots of unneeded calls to bcmp. (experimental.) 13) Significantly shrink, cleanup, and make slightly faster the vm_fault.c code. Use generation counts, get rid of unneded collpase operations, and clean up the cluster code. 14) Make vm_zone more suitable for TSM. This commit is partially as a result of discussions and contributions from other people, including DG, Tor Egge, PHK, and probably others that I have forgotten to attribute (so let me know, if I forgot.) This is not the infamous, final cleanup of the vnode stuff, but a necessary step. Vnode mgmt should be correct, but things might still change, and there is still some missing stuff (like ioopt, and physical backing of non-merged cache files, debugging of layering concepts.)
1998-01-22 17:30:44 +00:00
(vm_offset_t *)&kmemlimit, (vm_size_t)(npg * PAGE_SIZE));
Get rid of the ad-hoc memory allocator for vm_map_entries, in lieu of a simple, clean zone type allocator. This new allocator will also be used for machine dependent pmap PV entries.
1997-08-05 00:02:08 +00:00
kmem_map->system_map = 1;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
for (indx = 0; indx < MINBUCKET + 16; indx++) {
First pass at cleaning up macros relating to pages, clusters and all that.
1996-05-02 10:43:17 +00:00
if (1 << indx >= PAGE_SIZE)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
bucket[indx].kb_elmpercl = 1;
else
First pass at cleaning up macros relating to pages, clusters and all that.
1996-05-02 10:43:17 +00:00
bucket[indx].kb_elmpercl = PAGE_SIZE / (1 << indx);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
bucket[indx].kb_highwat = 5 * bucket[indx].kb_elmpercl;
}
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
}
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
void
malloc_init(data)
void *data;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
{
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
struct malloc_type *type = (struct malloc_type *)data;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
KAME netinet6 basic part(no IPsec,no V6 Multicast Forwarding, no UDP/TCP for IPv6 yet) With this patch, you can assigne IPv6 addr automatically, and can reply to IPv6 ping. Reviewed by: freebsd-arch, cvs-committers Obtained from: KAME project
1999-11-22 02:45:11 +00:00
if (type->ks_magic != M_MAGIC)
Remove the long description from the in-kernel datastructure. Put a magic field in there instead, to help catch uninitialized malloc types.
1997-10-28 19:01:02 +00:00
panic("malloc type lacks magic");
Fixed corruption of the kmemstatistcs list. The first malloc() with malloc type at the tail of the list changed the list from linear to circular. This seemed to cause surprisingly few problems, but it now causes weird output from `vmstat -m', probably because a more important malloc type is now at the tail of the list. Fix it by abusing ks_limit instead of ks_next as a flag for being on the list. Don't forget to clear the flag when a malloc type is uninit'ed. Uninit'ing is still fundamentally broken -- it loses history.
1999-05-12 11:11:27 +00:00
if (type->ks_limit != 0)
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
return;
Some fixes from John Hood: 1) Fix the initialization of malloc structure that changed due to perf opt. 2) Remove unneeded include. 3) An initialization assert added to malloc. Submitted by: John Hood <cgull@smoke.marlboro.vt.us>
1997-12-05 05:36:58 +00:00
if (cnt.v_page_count == 0)
panic("malloc_init not allowed before vm init");
Implement what I mentioned in rev 1.18: limit per-bucket allocations to 60% of physical memory or 60% of malloc area size, whichever is smaller.
1996-01-29 11:12:37 +00:00
/*
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
* The default limits for each malloc region is 1/2 of the
* malloc portion of the kmem map size.
Implement what I mentioned in rev 1.18: limit per-bucket allocations to 60% of physical memory or 60% of malloc area size, whichever is smaller.
1996-01-29 11:12:37 +00:00
*/
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
type->ks_limit = vm_kmem_size / 2;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
type->ks_next = kmemstatistics;
kmemstatistics = type;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
void
malloc_uninit(data)
void *data;
{
struct malloc_type *type = (struct malloc_type *)data;
struct malloc_type *t;
Move #ifdef to the right place.
2000-06-29 09:26:26 +00:00
#ifdef INVARIANTS
If kernel compiled with INVARIANTS: On unload, remove references from freelist to memory type defined by module. Print a warning if module defines and allocate its own memory type, but didn't free it all on unload. Reviewed by: peter
2000-06-29 03:41:30 +00:00
struct kmembuckets *kbp;
struct freelist *freep;
long indx;
int s;
#endif
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
KAME netinet6 basic part(no IPsec,no V6 Multicast Forwarding, no UDP/TCP for IPv6 yet) With this patch, you can assigne IPv6 addr automatically, and can reply to IPv6 ping. Reviewed by: freebsd-arch, cvs-committers Obtained from: KAME project
1999-11-22 02:45:11 +00:00
if (type->ks_magic != M_MAGIC)
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
panic("malloc type lacks magic");
if (cnt.v_page_count == 0)
panic("malloc_uninit not allowed before vm init");
Fixed corruption of the kmemstatistcs list. The first malloc() with malloc type at the tail of the list changed the list from linear to circular. This seemed to cause surprisingly few problems, but it now causes weird output from `vmstat -m', probably because a more important malloc type is now at the tail of the list. Fix it by abusing ks_limit instead of ks_next as a flag for being on the list. Don't forget to clear the flag when a malloc type is uninit'ed. Uninit'ing is still fundamentally broken -- it loses history.
1999-05-12 11:11:27 +00:00
if (type->ks_limit == 0)
panic("malloc_uninit on uninitialized type");
If kernel compiled with INVARIANTS: On unload, remove references from freelist to memory type defined by module. Print a warning if module defines and allocate its own memory type, but didn't free it all on unload. Reviewed by: peter
2000-06-29 03:41:30 +00:00
#ifdef INVARIANTS
s = splmem();
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_lock(&malloc_mtx);
If kernel compiled with INVARIANTS: On unload, remove references from freelist to memory type defined by module. Print a warning if module defines and allocate its own memory type, but didn't free it all on unload. Reviewed by: peter
2000-06-29 03:41:30 +00:00
for (indx = 0; indx < MINBUCKET + 16; indx++) {
kbp = bucket + indx;
freep = (struct freelist*)kbp->kb_next;
while (freep) {
if (freep->type == type)
freep->type = M_FREE;
freep = (struct freelist*)freep->next;
}
}
splx(s);
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
2001-02-09 06:11:45 +00:00
mtx_unlock(&malloc_mtx);
If kernel compiled with INVARIANTS: On unload, remove references from freelist to memory type defined by module. Print a warning if module defines and allocate its own memory type, but didn't free it all on unload. Reviewed by: peter
2000-06-29 03:41:30 +00:00
if (type->ks_memuse != 0)
printf("malloc_uninit: %ld bytes of '%s' still allocated\n",
type->ks_memuse, type->ks_shortdesc);
#endif
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
if (type == kmemstatistics)
kmemstatistics = type->ks_next;
else {
for (t = kmemstatistics; t->ks_next != NULL; t = t->ks_next) {
if (t->ks_next == type) {
t->ks_next = type->ks_next;
break;
}
}
}
Fixed corruption of the kmemstatistcs list. The first malloc() with malloc type at the tail of the list changed the list from linear to circular. This seemed to cause surprisingly few problems, but it now causes weird output from `vmstat -m', probably because a more important malloc type is now at the tail of the list. Fix it by abusing ks_limit instead of ks_next as a flag for being on the list. Don't forget to clear the flag when a malloc type is uninit'ed. Uninit'ing is still fundamentally broken -- it loses history.
1999-05-12 11:11:27 +00:00
type->ks_next = NULL;
type->ks_limit = 0;
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
}
Reference in New Issue Copy Permalink