d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4f240aff01
freebsd-skq/sys/kern/kern_malloc.c

715 lines
18 KiB
C
Raw Normal View History

BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Copyright (c) 1987, 1991, 1993
* The Regents of the University of California. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 4. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)kern_malloc.c 8.3 (Berkeley) 1/4/94
*/
Use __FBSDID().
2003-06-11 00:56:59 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#include "opt_vm.h"
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/param.h>
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
#include <sys/systm.h>
Update for the KDB framework: o Make debugging code conditional upon KDB instead of DDB. o Call kdb_enter() instead of Debugger(). o Call kdb_backtrace() instead of db_print_backtrace() or backtrace(). kern_mutex.c: o Replace checks for db_active with checks for kdb_active and make them unconditional. kern_shutdown.c: o s/DDB_UNATTENDED/KDB_UNATTENDED/g o s/DDB_TRACE/KDB_TRACE/g o Save the TID of the thread doing the kernel dump so the debugger knows which thread to select as the current when debugging the kernel core file. o Clear kdb_active instead of db_active and do so unconditionally. o Remove backtrace() implementation. kern_synch.c: o Call kdb_reenter() instead of db_error().
2004-07-10 21:36:01 +00:00
#include <sys/kdb.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/kernel.h>
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files. Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files. Sort sys/*.h includes where possible in affected files. OK'ed by: bde (with reservations)
2001-05-01 08:13:21 +00:00
#include <sys/lock.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/malloc.h>
Fixed two bugs in the calculation of the malloc area (kmem_map) size: 1) The calculation didn't account for NMBCLUSTERS, so if a large number of clusters was specified, it would leave little or no space for kernel malloc. 2) It was bogusly restricted to v_page_count. This doesn't take into account the sparseness of the malloc area and would have caused problems on machines with small amounts of memory. It should probably instead be changed to set the malloc limit to be constrained by the amount of memory, but I didn't do this.
1996-01-29 09:58:34 +00:00
#include <sys/mbuf.h>
- machine/mutex.h -> sys/mutex.h - Use MUTEX_DECLARE() and MTX_COLD for the malloc_mtx mutex
2000-10-20 07:29:16 +00:00
#include <sys/mutex.h>
Untangled the vm.h include file spaghetti.
1995-12-07 12:48:31 +00:00
#include <sys/vmmeter.h>
Make intr_nesting_level per-process, rather than per-cpu. Setup interrupt threads to run with it always >= 1, so that malloc can detect M_WAITOK from "interrupt" context. This is also necessary in order to context switch from sched_ithd() directly. Reviewed By: peter
2001-01-21 19:25:07 +00:00
#include <sys/proc.h>
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
#include <sys/sysctl.h>
Introduce malloc_last_fail() which returns the number of seconds since malloc(9) failed last time. This is intended to help code adjust memory usage to the current circumstances. A typical use could be: if (malloc_last_fail() < 60) reduce_cache_by_one();
2002-11-01 18:58:12 +00:00
#include <sys/time.h>
Don't #include <sys/proc.h>, since machine/mutex.h does it now.
2000-09-23 00:01:37 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <vm/vm.h>
- Split UMA_ZFLAG_OFFPAGE into UMA_ZFLAG_OFFPAGE and UMA_ZFLAG_HASH. - Remove all instances of the mallochash. - Stash the slab pointer in the vm page's object pointer when allocating from the kmem_obj. - Use the overloaded object pointer to find slabs for malloced memory.
2002-09-18 08:26:30 +00:00
#include <vm/pmap.h>
Untangled the vm.h include file spaghetti.
1995-12-07 12:48:31 +00:00
#include <vm/vm_param.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <vm/vm_kern.h>
Untangled the vm.h include file spaghetti.
1995-12-07 12:48:31 +00:00
#include <vm/vm_extern.h>
Get rid of the ad-hoc memory allocator for vm_map_entries, in lieu of a simple, clean zone type allocator. This new allocator will also be used for machine dependent pmap PV entries.
1997-08-05 00:02:08 +00:00
#include <vm/vm_map.h>
- Split UMA_ZFLAG_OFFPAGE into UMA_ZFLAG_OFFPAGE and UMA_ZFLAG_HASH. - Remove all instances of the mallochash. - Stash the slab pointer in the vm page's object pointer when allocating from the kmem_obj. - Use the overloaded object pointer to find slabs for malloced memory.
2002-09-18 08:26:30 +00:00
#include <vm/vm_page.h>
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
#include <vm/uma.h>
#include <vm/uma_int.h>
Add a new UMA debugging facility. This will overwrite freed memory with 0xdeadc0de and then check for it just before memory is handed off as part of a new request. This will catch any post free/pre alloc modification of memory, as well as introduce errors for anything that tries to dereference it as a pointer. This code takes the form of special init, fini, ctor and dtor routines that are specificly used by malloc. It is in a seperate file because additional debugging aids will want to live here as well.
2002-04-30 07:54:25 +00:00
#include <vm/uma_dbg.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
KASSERT that we cannot use M_WAITOK in interrupt context. Reviewed by: bde
1999-09-19 08:40:11 +00:00
#if defined(INVARIANTS) && defined(__i386__)
#include <machine/cpu.h>
#endif
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
/*
* When realloc() is called, if the new size is sufficiently smaller than
* the old size, realloc() will allocate a new, smaller block to avoid
* wasting memory. 'Sufficiently smaller' is defined as: newsize <=
* oldsize / 2^n, where REALLOC_FRACTION defines the value of 'n'.
*/
#ifndef REALLOC_FRACTION
#define REALLOC_FRACTION 1 /* new block if <= half the size */
#endif
Before we start to mess with the VFS name-cache clean things up a little bit: Isolate the namecache in its own file, and give it a dedicated malloc type.
1999-10-03 12:18:29 +00:00
MALLOC_DEFINE(M_CACHE, "cache", "Various Dynamically allocated caches");
Get rid of MALLOC_INSTANTIATE and MALLOC_MAKE_TYPE(). Just handle the 3 malloc types declared in <sys/malloc.h> like other global malloc types.
1999-09-11 16:41:39 +00:00
MALLOC_DEFINE(M_DEVBUF, "devbuf", "device driver memory");
MALLOC_DEFINE(M_TEMP, "temp", "misc temporary data buffers");
KAME netinet6 basic part(no IPsec,no V6 Multicast Forwarding, no UDP/TCP for IPv6 yet) With this patch, you can assigne IPv6 addr automatically, and can reply to IPv6 ping. Reviewed by: freebsd-arch, cvs-committers Obtained from: KAME project
1999-11-22 02:45:11 +00:00
MALLOC_DEFINE(M_IP6OPT, "ip6opt", "IPv6 options");
MALLOC_DEFINE(M_IP6NDP, "ip6ndp", "IPv6 Neighbor Discovery");
Remove __P.
2002-03-19 21:25:46 +00:00
static void kmeminit(void *);
Reviewed by: julian with quick glances by bruce and others Submitted by: terry (terry lambert) This is a composite of 3 patch sets submitted by terry. they are: New low-level init code that supports loadbal modules better some cleanups in the namei code to help terry in 16-bit character support some changes to the mount-root code to make it a little more modular.. NOTE: mounting root off cdrom or NFS MIGHT be broken as I haven't been able to test those cases.. certainly mounting root of disk still works just fine.. mfs should work but is untested. (tomorrows task) The low level init stuff includes a total rewrite of init_main.c to make it possible for new modules to have an init phase by simply adding an entry to a TEXT_SET (or is it DATA_SET) list. thus a new module can be added to the kernel without editing any other files other than the 'files' file.
1995-08-28 09:19:25 +00:00
SYSINIT(kmem, SI_SUB_KMEM, SI_ORDER_FIRST, kmeminit, NULL)
Last major round (Unless Bruce thinks of somthing :-) of malloc changes. Distribute all but the most fundamental malloc types. This time I also remembered the trick to making things static: Put "static" in front of them. A couple of finer points by: bde
1997-10-12 20:26:33 +00:00
static MALLOC_DEFINE(M_FREE, "free", "should be on free list");
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
static struct malloc_type *kmemstatistics;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
static char *kmembase;
Fixed staticization. buckets[] was staticized but was still declared extern in <sys/malloc.h> and it should not have been staticized for the !(KMEMSTATS || DIAGNOSTIC) case. Fixed the !(KMEMSTATS || DIAGNOSTIC) case. The MALLOC() and FREE() macros are evil, but code generally doesn't allow for this and some code involving else clauses did not compile. Finished staticization.
1997-09-16 13:52:04 +00:00
static char *kmemlimit;
Fix null-pointer dereference crash when the system is intentionally run out of KVM through a mmap()/fork() bomb that allocates hundreds of thousands of vm_map_entry structures. Add panic to make null-pointer dereference crash a little more verbose. Add a new sysctl, vm.max_proc_mmap, which specifies the maximum number of mmap()'d spaces (discrete vm_map_entry's in the process). The value defaults to around 9000 for a 128MB machine. The test is scaled for the number of processes sharing a vmspace (aka linux threads). Setting the value to 0 disables the feature. PR: kern/16573 Approved by: jkh
2000-02-16 21:11:33 +00:00
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
#define KMEM_ZSHIFT 4
#define KMEM_ZBASE 16
#define KMEM_ZMASK (KMEM_ZBASE - 1)
- Only use UMA to cache malloc requests up to PAGE_SIZE. Values larger than this are requested very infrequently and waste memory when we cache spares.
2003-09-19 04:39:08 +00:00
#define KMEM_ZMAX PAGE_SIZE
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
#define KMEM_ZSIZE (KMEM_ZMAX >> KMEM_ZSHIFT)
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
static u_int8_t kmemsize[KMEM_ZSIZE + 1];
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
/* These won't be powers of two for long */
struct {
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
int kz_size;
char *kz_name;
uma_zone_t kz_zone;
} kmemzones[] = {
{16, "16", NULL},
{32, "32", NULL},
{64, "64", NULL},
{128, "128", NULL},
{256, "256", NULL},
{512, "512", NULL},
{1024, "1024", NULL},
{2048, "2048", NULL},
{4096, "4096", NULL},
- Only use UMA to cache malloc requests up to PAGE_SIZE. Values larger than this are requested very infrequently and waste memory when we cache spares.
2003-09-19 04:39:08 +00:00
#if PAGE_SIZE > 4096
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
{8192, "8192", NULL},
- Only use UMA to cache malloc requests up to PAGE_SIZE. Values larger than this are requested very infrequently and waste memory when we cache spares.
2003-09-19 04:39:08 +00:00
#if PAGE_SIZE > 8192
Re-add the 16384 bucket also. Submitted by: green
2002-04-29 17:53:23 +00:00
{16384, "16384", NULL},
- Only use UMA to cache malloc requests up to PAGE_SIZE. Values larger than this are requested very infrequently and waste memory when we cache spares.
2003-09-19 04:39:08 +00:00
#if PAGE_SIZE > 16384
Revert a portion of kern_malloc.c:1.99, which (in addition to adding malloc profiling) also modified the set of pre-defined buckets for the memory allocator. For reasons unknown to me, this resulted in extensive memory corruption in the kernel, in particular on SMP boxes, so I'm committing this work-around until Jeff gets a chance to debug it properly. David Wolfskill pointed me at this commit as the one that might be a problem; I've been running this code on two dual-processor burn-in boxes for about 12 hours now, and the rate of panics due to memory corruption has dropped to zero (from one every five minutes). Hopefully not treading on the toes of: jeff
2002-04-29 17:12:02 +00:00
{32768, "32768", NULL},
- Only use UMA to cache malloc requests up to PAGE_SIZE. Values larger than this are requested very infrequently and waste memory when we cache spares.
2003-09-19 04:39:08 +00:00
#if PAGE_SIZE > 32768
Revert a portion of kern_malloc.c:1.99, which (in addition to adding malloc profiling) also modified the set of pre-defined buckets for the memory allocator. For reasons unknown to me, this resulted in extensive memory corruption in the kernel, in particular on SMP boxes, so I'm committing this work-around until Jeff gets a chance to debug it properly. David Wolfskill pointed me at this commit as the one that might be a problem; I've been running this code on two dual-processor burn-in boxes for about 12 hours now, and the rate of panics due to memory corruption has dropped to zero (from one every five minutes). Hopefully not treading on the toes of: jeff
2002-04-29 17:12:02 +00:00
{65536, "65536", NULL},
- Only use UMA to cache malloc requests up to PAGE_SIZE. Values larger than this are requested very infrequently and waste memory when we cache spares.
2003-09-19 04:39:08 +00:00
#if PAGE_SIZE > 65536
#error "Unsupported PAGE_SIZE"
#endif /* 65536 */
#endif /* 32768 */
#endif /* 16384 */
#endif /* 8192 */
#endif /* 4096 */
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
{0, NULL},
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
};
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
u_int vm_kmem_size;
Rename the kern.vm.kmem.size tunable to the more logical vm.kmem_size. To assure backward compatibility (conditional on !BURN_BRIDGES), look it up by its old name first, and log a warning (but accept the setting) if it was found. If both the old and new name are defined, the new name takes precedence. Also export vm.kmem_size as a read-only sysctl variable; I find it hard to tune a parameter when I don't know its default value, especially when that default value is computed at boot time.
2004-01-27 15:59:38 +00:00
SYSCTL_UINT(_vm, OID_AUTO, kmem_size, CTLFLAG_RD, &vm_kmem_size, 0,
"Size of kernel memory");
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
Turn VM_KMEM_SIZE_MAX and VM_KMEM_SIZE_SCALE into tunables. MFC after: 3 days
2004-09-29 14:21:40 +00:00
u_int vm_kmem_size_max;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_max, CTLFLAG_RD, &vm_kmem_size_max, 0,
"Maximum size of kernel memory");
u_int vm_kmem_size_scale;
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_scale, CTLFLAG_RD, &vm_kmem_size_scale, 0,
"Scale factor for kernel memory size");
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
/*
- Split UMA_ZFLAG_OFFPAGE into UMA_ZFLAG_OFFPAGE and UMA_ZFLAG_HASH. - Remove all instances of the mallochash. - Stash the slab pointer in the vm page's object pointer when allocating from the kmem_obj. - Use the overloaded object pointer to find slabs for malloced memory.
2002-09-18 08:26:30 +00:00
* The malloc_mtx protects the kmemstatistics linked list.
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
*/
struct mtx malloc_mtx;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
Finish adding support code for sysctl kern.mprof. This dumps some malloc information related to bucket size effeciency. Three things are printed on each row: Size is the size the user actually asked for rounded to 16 bytes. Requests is the number of times this size was asked for. Real Size is the size we actually handed out. At the end the total memory used and total waste is displayed. Currently my system displays about 33% wasted memory. The intent of this code is to gather statistics for tuning the malloc bucket sizes. It is not intended to be run with INVARIANTS and it is not entirely mp safe. It can be enabled via 'options MALLOC_PROFILE' which was commited earlier.
2002-04-15 05:24:01 +00:00
#ifdef MALLOC_PROFILE
uint64_t krequests[KMEM_ZSIZE + 1];
static int sysctl_kern_mprof(SYSCTL_HANDLER_ARGS);
#endif
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
Finish adding support code for sysctl kern.mprof. This dumps some malloc information related to bucket size effeciency. Three things are printed on each row: Size is the size the user actually asked for rounded to 16 bytes. Requests is the number of times this size was asked for. Real Size is the size we actually handed out. At the end the total memory used and total waste is displayed. Currently my system displays about 33% wasted memory. The intent of this code is to gather statistics for tuning the malloc bucket sizes. It is not intended to be run with INVARIANTS and it is not entirely mp safe. It can be enabled via 'options MALLOC_PROFILE' which was commited earlier.
2002-04-15 05:24:01 +00:00
static int sysctl_kern_malloc(SYSCTL_HANDLER_ARGS);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
Introduce malloc_last_fail() which returns the number of seconds since malloc(9) failed last time. This is intended to help code adjust memory usage to the current circumstances. A typical use could be: if (malloc_last_fail() < 60) reduce_cache_by_one();
2002-11-01 18:58:12 +00:00
/* time_uptime of last malloc(9) failure */
static time_t t_malloc_fail;
Add a new kernel option, MALLOC_MAKE_FAILURES, which compiles in a debugging feature causing M_NOWAIT allocations to fail at a specified rate. This can be useful for detecting poor handling of M_NOWAIT: the most frequent problems I've bumped into are unconditional deference of the pointer even though it's NULL, and hangs as a result of a lost event where memory for the event couldn't be allocated. Two sysctls are added: debug.malloc.failure_rate How often to generate a failure: if set to 0 (default), this feature is disabled. Otherwise, the frequency of failures -- I've been using 10 (one in ten mallocs fails), but other popular settings might be much lower or much higher. debug.malloc.failure_count Number of times a coerced malloc failure has occurred as a result of this feature. Useful for tracking what might have happened and whether failures are being generated. Useful possible additions: tying failure rate to malloc type, printfs indicating the thread that experienced the coerced failure. Reviewed by: jeffr, jhb
2003-03-26 20:18:40 +00:00
#ifdef MALLOC_MAKE_FAILURES
/*
* Causes malloc failures every (n) mallocs with M_NOWAIT. If set to 0,
* doesn't cause failures.
*/
SYSCTL_NODE(_debug, OID_AUTO, malloc, CTLFLAG_RD, 0,
"Kernel malloc debugging options");
static int malloc_failure_rate;
static int malloc_nowait_count;
static int malloc_failure_count;
SYSCTL_INT(_debug_malloc, OID_AUTO, failure_rate, CTLFLAG_RW,
&malloc_failure_rate, 0, "Every (n) mallocs with M_NOWAIT will fail");
Permit debug.malloc.failure_rate to be specified using a tunable so that the feature can be enabled during the boot process. Note the continued limitation that FreeBSD fails so rapidly with this setting enabled that it's hard to narrow down particular failures for correction; we really need per-malloc type failure rates.
2003-03-26 20:44:29 +00:00
TUNABLE_INT("debug.malloc.failure_rate", &malloc_failure_rate);
Add a new kernel option, MALLOC_MAKE_FAILURES, which compiles in a debugging feature causing M_NOWAIT allocations to fail at a specified rate. This can be useful for detecting poor handling of M_NOWAIT: the most frequent problems I've bumped into are unconditional deference of the pointer even though it's NULL, and hangs as a result of a lost event where memory for the event couldn't be allocated. Two sysctls are added: debug.malloc.failure_rate How often to generate a failure: if set to 0 (default), this feature is disabled. Otherwise, the frequency of failures -- I've been using 10 (one in ten mallocs fails), but other popular settings might be much lower or much higher. debug.malloc.failure_count Number of times a coerced malloc failure has occurred as a result of this feature. Useful for tracking what might have happened and whether failures are being generated. Useful possible additions: tying failure rate to malloc type, printfs indicating the thread that experienced the coerced failure. Reviewed by: jeffr, jhb
2003-03-26 20:18:40 +00:00
SYSCTL_INT(_debug_malloc, OID_AUTO, failure_count, CTLFLAG_RD,
&malloc_failure_count, 0, "Number of imposed M_NOWAIT malloc failures");
#endif
Introduce malloc_last_fail() which returns the number of seconds since malloc(9) failed last time. This is intended to help code adjust memory usage to the current circumstances. A typical use could be: if (malloc_last_fail() < 60) reduce_cache_by_one();
2002-11-01 18:58:12 +00:00
int
malloc_last_fail(void)
{
return (time_uptime - t_malloc_fail);
}
Reimplement contigmalloc(9) with an algorithm which stands a greatly- improved chance of working despite pressure from running programs. Instead of trying to throw a bunch of pages out to swap and hope for the best, only a range that can potentially fulfill contigmalloc(9)'s request will have its contents paged out (potentially, not forcibly) at a time. The new contigmalloc operation still operates in three passes, but it could potentially be tuned to more or less. The first pass only looks at pages in the cache and free pages, so they would be thrown out without having to block. If this is not enough, the subsequent passes page out any unwired memory. To combat memory pressure refragmenting the section of memory being laundered, each page is removed from the systems' free memory queue once it has been freed so that blocking later doesn't cause the memory laundered so far to get reallocated. The page-out operations are now blocking, as it would make little sense to try to push out a page, then get its status immediately afterward to remove it from the available free pages queue, if it's unlikely to have been freed. Another change is that if KVA allocation fails, the allocated memory segment will be freed and not leaked. There is a sysctl/tunable, defaulting to on, which causes the old contigmalloc() algorithm to be used. Nonetheless, I have been using vm.old_contigmalloc=0 for over a month. It is safe to switch at run-time to see the difference it makes. A new interface has been used which does not require mapping the allocated pages into KVA: vm_page.h functions vm_page_alloc_contig() and vm_page_release_contig(). These are what vm.old_contigmalloc=0 uses internally, so the sysctl/tunable does not affect their operation. When using the contigmalloc(9) and contigfree(9) interfaces, memory is now tracked with malloc(9) stats. Several functions have been exported from kern_malloc.c to allow other subsystems to use these statistics, as well. This invalidates the BUGS section of the contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
/*
* Add this to the informational malloc_type bucket.
*/
static void
malloc_type_zone_allocated(struct malloc_type *ksp, unsigned long size,
int zindx)
{
mtx_lock(&ksp->ks_mtx);
ksp->ks_calls++;
if (zindx != -1)
ksp->ks_size |= 1 << zindx;
if (size != 0) {
ksp->ks_memuse += size;
ksp->ks_inuse++;
if (ksp->ks_memuse > ksp->ks_maxused)
ksp->ks_maxused = ksp->ks_memuse;
}
mtx_unlock(&ksp->ks_mtx);
}
void
malloc_type_allocated(struct malloc_type *ksp, unsigned long size)
{
malloc_type_zone_allocated(ksp, size, -1);
}
/*
* Remove this allocation from the informational malloc_type bucket.
*/
void
malloc_type_freed(struct malloc_type *ksp, unsigned long size)
{
mtx_lock(&ksp->ks_mtx);
KASSERT(size <= ksp->ks_memuse,
("malloc(9)/free(9) confusion.\n%s",
"Probably freeing with wrong type, but maybe not here."));
ksp->ks_memuse -= size;
ksp->ks_inuse--;
mtx_unlock(&ksp->ks_mtx);
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
This is a rather large commit that encompasses the new swapper, changes to the VM system to support the new swapper, VM bug fixes, several VM optimizations, and some additional revamping of the VM code. The specific bug fixes will be documented with additional forced commits. This commit is somewhat rough in regards to code cleanup issues. Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>
1999-01-21 08:29:12 +00:00
* malloc:
*
* Allocate a block of memory.
*
* If M_NOWAIT is set, this routine will not block and return NULL if
* the allocation fails.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
void *
malloc(size, type, flags)
unsigned long size;
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
struct malloc_type *type;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
int flags;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
int indx;
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
caddr_t va;
uma_zone_t zone;
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
uma_keg_t keg;
Under #ifdef DIAGNOSTIC, fill malloc(9) allocations which do not have M_ZERO specified with 0x70. (malloc_flags=J for the kernel :-)
2003-02-01 10:07:49 +00:00
#ifdef DIAGNOSTIC
unsigned long osize = size;
#endif
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
PHCC[1]: I had commented the #ifdef INVARIANTS checks out to make sure I ran this code in all kernels and forgot to comment the #ifdefs back in before I committed. Spotted by: bmilekic [1] PHCC = Pointy Hat Correction Commit
2003-03-10 20:24:54 +00:00
#ifdef INVARIANTS
Make malloc and mbuf allocation mode flags nonoverlapping. Under INVARIANTS whine if we get incompatible flags. Submitted by: imp
2003-03-10 19:39:53 +00:00
/*
* To make sure that WAITOK or NOWAIT is set, but not more than
* one, and check against the API botches that are common.
*/
indx = flags & (M_WAITOK | M_NOWAIT | M_DONTWAIT | M_TRYWAIT);
if (indx != M_NOWAIT && indx != M_WAITOK) {
static struct timeval lasterr;
static int curerr, once;
if (once == 0 && ppsratecheck(&lasterr, &curerr, 1)) {
printf("Bad malloc flags: %x\n", indx);
Update for the KDB framework: o Make debugging code conditional upon KDB instead of DDB. o Call kdb_enter() instead of Debugger(). o Call kdb_backtrace() instead of db_print_backtrace() or backtrace(). kern_mutex.c: o Replace checks for db_active with checks for kdb_active and make them unconditional. kern_shutdown.c: o s/DDB_UNATTENDED/KDB_UNATTENDED/g o s/DDB_TRACE/KDB_TRACE/g o Save the TID of the thread doing the kernel dump so the debugger knows which thread to select as the current when debugging the kernel core file. o Clear kdb_active instead of db_active and do so unconditionally. o Remove backtrace() implementation. kern_synch.c: o Call kdb_reenter() instead of db_error().
2004-07-10 21:36:01 +00:00
kdb_backtrace();
Make malloc and mbuf allocation mode flags nonoverlapping. Under INVARIANTS whine if we get incompatible flags. Submitted by: imp
2003-03-10 19:39:53 +00:00
flags |= M_WAITOK;
once++;
}
}
PHCC[1]: I had commented the #ifdef INVARIANTS checks out to make sure I ran this code in all kernels and forgot to comment the #ifdefs back in before I committed. Spotted by: bmilekic [1] PHCC = Pointy Hat Correction Commit
2003-03-10 20:24:54 +00:00
#endif
Add a basic sanity check on pointers passed to free(9). Should be improved by: jeff
2002-04-23 18:50:25 +00:00
#if 0
if (size == 0)
Update for the KDB framework: o Make debugging code conditional upon KDB instead of DDB. o Call kdb_enter() instead of Debugger(). o Call kdb_backtrace() instead of db_print_backtrace() or backtrace(). kern_mutex.c: o Replace checks for db_active with checks for kdb_active and make them unconditional. kern_shutdown.c: o s/DDB_UNATTENDED/KDB_UNATTENDED/g o s/DDB_TRACE/KDB_TRACE/g o Save the TID of the thread doing the kernel dump so the debugger knows which thread to select as the current when debugging the kernel core file. o Clear kdb_active instead of db_active and do so unconditionally. o Remove backtrace() implementation. kern_synch.c: o Call kdb_reenter() instead of db_error().
2004-07-10 21:36:01 +00:00
kdb_enter("zero size malloc");
Add a new kernel option, MALLOC_MAKE_FAILURES, which compiles in a debugging feature causing M_NOWAIT allocations to fail at a specified rate. This can be useful for detecting poor handling of M_NOWAIT: the most frequent problems I've bumped into are unconditional deference of the pointer even though it's NULL, and hangs as a result of a lost event where memory for the event couldn't be allocated. Two sysctls are added: debug.malloc.failure_rate How often to generate a failure: if set to 0 (default), this feature is disabled. Otherwise, the frequency of failures -- I've been using 10 (one in ten mallocs fails), but other popular settings might be much lower or much higher. debug.malloc.failure_count Number of times a coerced malloc failure has occurred as a result of this feature. Useful for tracking what might have happened and whether failures are being generated. Useful possible additions: tying failure rate to malloc type, printfs indicating the thread that experienced the coerced failure. Reviewed by: jeffr, jhb
2003-03-26 20:18:40 +00:00
#endif
#ifdef MALLOC_MAKE_FAILURES
if ((flags & M_NOWAIT) && (malloc_failure_rate != 0)) {
atomic_add_int(&malloc_nowait_count, 1);
if ((malloc_nowait_count % malloc_failure_rate) == 0) {
atomic_add_int(&malloc_failure_count, 1);
Update the "last malloc failure timestamp" also for simulated malloc errors.
2003-04-25 21:49:24 +00:00
t_malloc_fail = time_uptime;
Add a new kernel option, MALLOC_MAKE_FAILURES, which compiles in a debugging feature causing M_NOWAIT allocations to fail at a specified rate. This can be useful for detecting poor handling of M_NOWAIT: the most frequent problems I've bumped into are unconditional deference of the pointer even though it's NULL, and hangs as a result of a lost event where memory for the event couldn't be allocated. Two sysctls are added: debug.malloc.failure_rate How often to generate a failure: if set to 0 (default), this feature is disabled. Otherwise, the frequency of failures -- I've been using 10 (one in ten mallocs fails), but other popular settings might be much lower or much higher. debug.malloc.failure_count Number of times a coerced malloc failure has occurred as a result of this feature. Useful for tracking what might have happened and whether failures are being generated. Useful possible additions: tying failure rate to malloc type, printfs indicating the thread that experienced the coerced failure. Reviewed by: jeffr, jhb
2003-03-26 20:18:40 +00:00
return (NULL);
}
}
Add a basic sanity check on pointers passed to free(9). Should be improved by: jeff
2002-04-23 18:50:25 +00:00
#endif
Make malloc and mbuf allocation mode flags nonoverlapping. Under INVARIANTS whine if we get incompatible flags. Submitted by: imp
2003-03-10 19:39:53 +00:00
if (flags & M_WAITOK)
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process. Sorry john! (your next MFC will be a doosie!) Reviewed by: peter@freebsd.org, dillon@freebsd.org X-MFC after: ha ha ha ha
2001-09-12 08:38:13 +00:00
KASSERT(curthread->td_intr_nesting_level == 0,
Back out M_* changes, per decision of the TRB. Approved by: trb
2003-02-19 05:47:46 +00:00
("malloc(M_WAITOK) in interrupt context"));
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
if (size <= KMEM_ZMAX) {
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
if (size & KMEM_ZMASK)
size = (size & ~KMEM_ZMASK) + KMEM_ZBASE;
indx = kmemsize[size >> KMEM_ZSHIFT];
zone = kmemzones[indx].kz_zone;
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
keg = zone->uz_keg;
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
#ifdef MALLOC_PROFILE
krequests[size >> KMEM_ZSHIFT]++;
#endif
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
va = uma_zalloc(zone, flags);
Reimplement contigmalloc(9) with an algorithm which stands a greatly- improved chance of working despite pressure from running programs. Instead of trying to throw a bunch of pages out to swap and hope for the best, only a range that can potentially fulfill contigmalloc(9)'s request will have its contents paged out (potentially, not forcibly) at a time. The new contigmalloc operation still operates in three passes, but it could potentially be tuned to more or less. The first pass only looks at pages in the cache and free pages, so they would be thrown out without having to block. If this is not enough, the subsequent passes page out any unwired memory. To combat memory pressure refragmenting the section of memory being laundered, each page is removed from the systems' free memory queue once it has been freed so that blocking later doesn't cause the memory laundered so far to get reallocated. The page-out operations are now blocking, as it would make little sense to try to push out a page, then get its status immediately afterward to remove it from the available free pages queue, if it's unlikely to have been freed. Another change is that if KVA allocation fails, the allocated memory segment will be freed and not leaked. There is a sysctl/tunable, defaulting to on, which causes the old contigmalloc() algorithm to be used. Nonetheless, I have been using vm.old_contigmalloc=0 for over a month. It is safe to switch at run-time to see the difference it makes. A new interface has been used which does not require mapping the allocated pages into KVA: vm_page.h functions vm_page_alloc_contig() and vm_page_release_contig(). These are what vm.old_contigmalloc=0 uses internally, so the sysctl/tunable does not affect their operation. When using the contigmalloc(9) and contigfree(9) interfaces, memory is now tracked with malloc(9) stats. Several functions have been exported from kern_malloc.c to allow other subsystems to use these statistics, as well. This invalidates the BUGS section of the contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
if (va != NULL)
size = keg->uk_size;
malloc_type_zone_allocated(type, va == NULL ? 0 : size, indx);
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
} else {
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
size = roundup(size, PAGE_SIZE);
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
zone = NULL;
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
keg = NULL;
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
va = uma_large_malloc(size, flags);
Reimplement contigmalloc(9) with an algorithm which stands a greatly- improved chance of working despite pressure from running programs. Instead of trying to throw a bunch of pages out to swap and hope for the best, only a range that can potentially fulfill contigmalloc(9)'s request will have its contents paged out (potentially, not forcibly) at a time. The new contigmalloc operation still operates in three passes, but it could potentially be tuned to more or less. The first pass only looks at pages in the cache and free pages, so they would be thrown out without having to block. If this is not enough, the subsequent passes page out any unwired memory. To combat memory pressure refragmenting the section of memory being laundered, each page is removed from the systems' free memory queue once it has been freed so that blocking later doesn't cause the memory laundered so far to get reallocated. The page-out operations are now blocking, as it would make little sense to try to push out a page, then get its status immediately afterward to remove it from the available free pages queue, if it's unlikely to have been freed. Another change is that if KVA allocation fails, the allocated memory segment will be freed and not leaked. There is a sysctl/tunable, defaulting to on, which causes the old contigmalloc() algorithm to be used. Nonetheless, I have been using vm.old_contigmalloc=0 for over a month. It is safe to switch at run-time to see the difference it makes. A new interface has been used which does not require mapping the allocated pages into KVA: vm_page.h functions vm_page_alloc_contig() and vm_page_release_contig(). These are what vm.old_contigmalloc=0 uses internally, so the sysctl/tunable does not affect their operation. When using the contigmalloc(9) and contigfree(9) interfaces, memory is now tracked with malloc(9) stats. Several functions have been exported from kern_malloc.c to allow other subsystems to use these statistics, as well. This invalidates the BUGS section of the contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
malloc_type_allocated(type, va == NULL ? 0 : size);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Don't pass NULL pointer to memset if we are compiled with DIAGNOSTIC Approved by: re/rwatson
2003-05-12 05:09:56 +00:00
if (flags & M_WAITOK)
Back out M_* changes, per decision of the TRB. Approved by: trb
2003-02-19 05:47:46 +00:00
KASSERT(va != NULL, ("malloc(M_WAITOK) returned NULL"));
Don't pass NULL pointer to memset if we are compiled with DIAGNOSTIC Approved by: re/rwatson
2003-05-12 05:09:56 +00:00
else if (va == NULL)
Introduce malloc_last_fail() which returns the number of seconds since malloc(9) failed last time. This is intended to help code adjust memory usage to the current circumstances. A typical use could be: if (malloc_last_fail() < 60) reduce_cache_by_one();
2002-11-01 18:58:12 +00:00
t_malloc_fail = time_uptime;
Under #ifdef DIAGNOSTIC, fill malloc(9) allocations which do not have M_ZERO specified with 0x70. (malloc_flags=J for the kernel :-)
2003-02-01 10:07:49 +00:00
#ifdef DIAGNOSTIC
Don't pass NULL pointer to memset if we are compiled with DIAGNOSTIC Approved by: re/rwatson
2003-05-12 05:09:56 +00:00
if (va != NULL && !(flags & M_ZERO)) {
Under #ifdef DIAGNOSTIC, fill malloc(9) allocations which do not have M_ZERO specified with 0x70. (malloc_flags=J for the kernel :-)
2003-02-01 10:07:49 +00:00
memset(va, 0x70, osize);
}
#endif
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
return ((void *) va);
}
/*
This is a rather large commit that encompasses the new swapper, changes to the VM system to support the new swapper, VM bug fixes, several VM optimizations, and some additional revamping of the VM code. The specific bug fixes will be documented with additional forced commits. This commit is somewhat rough in regards to code cleanup issues. Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>
1999-01-21 08:29:12 +00:00
* free:
*
* Free a block of memory allocated by malloc.
*
* This routine may not block.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
void
Revert stuff which accidentally ended up in the previous commit.
2003-07-22 10:36:36 +00:00
free(addr, type)
void *addr;
Rename "struct kmemstats" to "struct malloc_type" it makes more sense now. Fix type argument to hashinit() and phashinit()
1997-10-10 18:14:23 +00:00
struct malloc_type *type;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
uma_slab_t slab;
u_long size;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
/* free(NULL, ...) does nothing */
if (addr == NULL)
return;
Reimplement contigmalloc(9) with an algorithm which stands a greatly- improved chance of working despite pressure from running programs. Instead of trying to throw a bunch of pages out to swap and hope for the best, only a range that can potentially fulfill contigmalloc(9)'s request will have its contents paged out (potentially, not forcibly) at a time. The new contigmalloc operation still operates in three passes, but it could potentially be tuned to more or less. The first pass only looks at pages in the cache and free pages, so they would be thrown out without having to block. If this is not enough, the subsequent passes page out any unwired memory. To combat memory pressure refragmenting the section of memory being laundered, each page is removed from the systems' free memory queue once it has been freed so that blocking later doesn't cause the memory laundered so far to get reallocated. The page-out operations are now blocking, as it would make little sense to try to push out a page, then get its status immediately afterward to remove it from the available free pages queue, if it's unlikely to have been freed. Another change is that if KVA allocation fails, the allocated memory segment will be freed and not leaked. There is a sysctl/tunable, defaulting to on, which causes the old contigmalloc() algorithm to be used. Nonetheless, I have been using vm.old_contigmalloc=0 for over a month. It is safe to switch at run-time to see the difference it makes. A new interface has been used which does not require mapping the allocated pages into KVA: vm_page.h functions vm_page_alloc_contig() and vm_page_release_contig(). These are what vm.old_contigmalloc=0 uses internally, so the sysctl/tunable does not affect their operation. When using the contigmalloc(9) and contigfree(9) interfaces, memory is now tracked with malloc(9) stats. Several functions have been exported from kern_malloc.c to allow other subsystems to use these statistics, as well. This invalidates the BUGS section of the contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
KASSERT(type->ks_memuse > 0,
Add two KASSERTS which trigger if free(9) would drag the "memuse" statistic for a malloc bucket under zero. This typically happens if you malloc(9) from one bucket and free to another.
2003-05-05 08:32:53 +00:00
("malloc(9)/free(9) confusion.\n%s",
"Probably freeing with wrong type, but maybe not here."));
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
size = 0;
- Split UMA_ZFLAG_OFFPAGE into UMA_ZFLAG_OFFPAGE and UMA_ZFLAG_HASH. - Remove all instances of the mallochash. - Stash the slab pointer in the vm page's object pointer when allocating from the kmem_obj. - Use the overloaded object pointer to find slabs for malloced memory.
2002-09-18 08:26:30 +00:00
slab = vtoslab((vm_offset_t)addr & (~UMA_SLAB_MASK));
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
if (slab == NULL)
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
panic("free: address %p(%p) has not been allocated.\n",
- Split UMA_ZFLAG_OFFPAGE into UMA_ZFLAG_OFFPAGE and UMA_ZFLAG_HASH. - Remove all instances of the mallochash. - Stash the slab pointer in the vm page's object pointer when allocating from the kmem_obj. - Use the overloaded object pointer to find slabs for malloced memory.
2002-09-18 08:26:30 +00:00
addr, (void *)((u_long)addr & (~UMA_SLAB_MASK)));
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
if (!(slab->us_flags & UMA_SLAB_MALLOC)) {
Hide a pointer to the malloc_type bucket at the end of the freed memory. If this memory is modified after it has been freed we can now report it's previous owner.
2002-05-02 09:07:04 +00:00
#ifdef INVARIANTS
struct malloc_type **mtp = addr;
#endif
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
size = slab->us_keg->uk_size;
Hide a pointer to the malloc_type bucket at the end of the freed memory. If this memory is modified after it has been freed we can now report it's previous owner.
2002-05-02 09:07:04 +00:00
#ifdef INVARIANTS
/*
* Cache a pointer to the malloc_type that most recently freed
* this memory here. This way we know who is most likely to
* have stepped on it later.
*
* This code assumes that size is a multiple of 8 bytes for
* 64 bit machines
*/
mtp = (struct malloc_type **)
((unsigned long)mtp & ~UMA_ALIGN_PTR);
mtp += (size - sizeof(struct malloc_type *)) /
sizeof(struct malloc_type *);
*mtp = type;
#endif
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
uma_zfree_arg(LIST_FIRST(&slab->us_keg->uk_zones), addr, slab);
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
} else {
size = slab->us_size;
uma_large_free(slab);
Add sufficient braces to keep egcs happy about potentially ambiguous if/else nesting.
1999-05-06 18:13:11 +00:00
}
Reimplement contigmalloc(9) with an algorithm which stands a greatly- improved chance of working despite pressure from running programs. Instead of trying to throw a bunch of pages out to swap and hope for the best, only a range that can potentially fulfill contigmalloc(9)'s request will have its contents paged out (potentially, not forcibly) at a time. The new contigmalloc operation still operates in three passes, but it could potentially be tuned to more or less. The first pass only looks at pages in the cache and free pages, so they would be thrown out without having to block. If this is not enough, the subsequent passes page out any unwired memory. To combat memory pressure refragmenting the section of memory being laundered, each page is removed from the systems' free memory queue once it has been freed so that blocking later doesn't cause the memory laundered so far to get reallocated. The page-out operations are now blocking, as it would make little sense to try to push out a page, then get its status immediately afterward to remove it from the available free pages queue, if it's unlikely to have been freed. Another change is that if KVA allocation fails, the allocated memory segment will be freed and not leaked. There is a sysctl/tunable, defaulting to on, which causes the old contigmalloc() algorithm to be used. Nonetheless, I have been using vm.old_contigmalloc=0 for over a month. It is safe to switch at run-time to see the difference it makes. A new interface has been used which does not require mapping the allocated pages into KVA: vm_page.h functions vm_page_alloc_contig() and vm_page_release_contig(). These are what vm.old_contigmalloc=0 uses internally, so the sysctl/tunable does not affect their operation. When using the contigmalloc(9) and contigfree(9) interfaces, memory is now tracked with malloc(9) stats. Several functions have been exported from kern_malloc.c to allow other subsystems to use these statistics, as well. This invalidates the BUGS section of the contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
malloc_type_freed(type, size);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
/*
* realloc: change the size of a memory block
*/
void *
realloc(addr, size, type, flags)
void *addr;
unsigned long size;
struct malloc_type *type;
int flags;
{
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
uma_slab_t slab;
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
unsigned long alloc;
void *newaddr;
/* realloc(NULL, ...) is equivalent to malloc(...) */
if (addr == NULL)
return (malloc(size, type, flags));
- Split UMA_ZFLAG_OFFPAGE into UMA_ZFLAG_OFFPAGE and UMA_ZFLAG_HASH. - Remove all instances of the mallochash. - Stash the slab pointer in the vm page's object pointer when allocating from the kmem_obj. - Use the overloaded object pointer to find slabs for malloced memory.
2002-09-18 08:26:30 +00:00
slab = vtoslab((vm_offset_t)addr & ~(UMA_SLAB_MASK));
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
/* Sanity check */
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
KASSERT(slab != NULL,
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
("realloc: address %p out of range", (void *)addr));
/* Get the size of the original block */
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
if (slab->us_keg)
alloc = slab->us_keg->uk_size;
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
else
alloc = slab->us_size;
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
/* Reuse the original block if appropriate */
if (size <= alloc
&& (size > (alloc >> REALLOC_FRACTION) || alloc == MINALLOCSIZE))
return (addr);
/* Allocate a new, bigger (or smaller) block */
if ((newaddr = malloc(size, type, flags)) == NULL)
return (NULL);
/* Copy over original contents */
bcopy(addr, newaddr, min(size, alloc));
free(addr, type);
return (newaddr);
}
/*
* reallocf: same as realloc() but free memory on failure.
*/
void *
reallocf(addr, size, type, flags)
void *addr;
unsigned long size;
struct malloc_type *type;
int flags;
{
void *mem;
Revert stuff which accidentally ended up in the previous commit.
2003-07-22 10:36:36 +00:00
if ((mem = realloc(addr, size, type, flags)) == NULL)
Add realloc() and reallocf(), and make free(NULL, ...) acceptable. Reviewed by: alfred
2002-03-13 01:42:33 +00:00
free(addr, type);
return (mem);
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Initialize the kernel memory allocator
*/
Reviewed by: julian with quick glances by bruce and others Submitted by: terry (terry lambert) This is a composite of 3 patch sets submitted by terry. they are: New low-level init code that supports loadbal modules better some cleanups in the namei code to help terry in 16-bit character support some changes to the mount-root code to make it a little more modular.. NOTE: mounting root off cdrom or NFS MIGHT be broken as I haven't been able to test those cases.. certainly mounting root of disk still works just fine.. mfs should work but is untested. (tomorrows task) The low level init stuff includes a total rewrite of init_main.c to make it possible for new modules to have an init phase by simply adding an entry to a TEXT_SET (or is it DATA_SET) list. thus a new module can be added to the kernel without editing any other files other than the 'files' file.
1995-08-28 09:19:25 +00:00
/* ARGSUSED*/
static void
Finished (?) cleaning up sysinit stuff.
1995-12-02 17:11:20 +00:00
kmeminit(dummy)
void *dummy;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
u_int8_t indx;
Fixed sign and overflow bugs that caused the allocation size of the kernel malloc region (kmem_map) to be wrong and semi-random on systems with more than 1GB of RAM. This is not a complete fix, but is sufficient for machines with 4GB or less of memory. A complete fix will require some changes to the getenv stuff so that 64bit values can be passed around. NOT FIXED: machines with more than 4GB of RAM (e.g. some large Alphas) since we're still using ints to hold some of the values. Reviewed by: bde
2000-01-28 04:04:58 +00:00
u_long mem_size;
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
int i;
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
Change callers of mtx_init() to pass in an appropriate lock type name. In most cases NULL is passed, but in some cases such as network driver locks (which use the MTX_NETWORK_LOCK macro) and UMA zone locks, a name is used. Tested on: i386, alpha, sparc64
2002-04-04 21:03:38 +00:00
mtx_init(&malloc_mtx, "malloc", NULL, MTX_DEF);
Add malloc_mtx to protect malloc and friends, so that they're thread-safe. Reviewed by: peter
2000-09-11 02:32:30 +00:00
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
/*
* Try to auto-tune the kernel memory size, so that it is
* more applicable for a wider range of machine sizes.
* On an X86, a VM_KMEM_SIZE_SCALE value of 4 is good, while
* a VM_KMEM_SIZE of 12MB is a fair compromise. The
* VM_KMEM_SIZE_MAX is dependent on the maximum KVA space
* available, and on an X86 with a total KVA space of 256MB,
* try to keep VM_KMEM_SIZE_MAX at 80MB or below.
*
* Note that the kmem_map is also used by the zone allocator,
* so make sure that there is enough space.
*/
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE;
Don't overflow when calculating vm_kmem_size. This fixes kmem_map too small panics on PAE machines which have odd > 4GB sizes (4.5 gig would render a 20MB of KVA for kmem_map instead of 200MB). Submitted by: John Cagle <john.cagle@hp.com>, jeff Reviewed by: jeff, peter, scottl, lots of USENIX folks
2003-06-11 05:18:59 +00:00
mem_size = cnt.v_page_count;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#if defined(VM_KMEM_SIZE_SCALE)
Turn VM_KMEM_SIZE_MAX and VM_KMEM_SIZE_SCALE into tunables. MFC after: 3 days
2004-09-29 14:21:40 +00:00
vm_kmem_size_scale = VM_KMEM_SIZE_SCALE;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#endif
Turn VM_KMEM_SIZE_MAX and VM_KMEM_SIZE_SCALE into tunables. MFC after: 3 days
2004-09-29 14:21:40 +00:00
TUNABLE_INT_FETCH("vm.kmem_size_scale", &vm_kmem_size_scale);
if (vm_kmem_size_scale > 0 &&
(mem_size / vm_kmem_size_scale) > (vm_kmem_size / PAGE_SIZE))
vm_kmem_size = (mem_size / vm_kmem_size_scale) * PAGE_SIZE;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#if defined(VM_KMEM_SIZE_MAX)
Turn VM_KMEM_SIZE_MAX and VM_KMEM_SIZE_SCALE into tunables. MFC after: 3 days
2004-09-29 14:21:40 +00:00
vm_kmem_size_max = VM_KMEM_SIZE_MAX;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
#endif
Turn VM_KMEM_SIZE_MAX and VM_KMEM_SIZE_SCALE into tunables. MFC after: 3 days
2004-09-29 14:21:40 +00:00
TUNABLE_INT_FETCH("vm.kmem_size_max", &vm_kmem_size_max);
if (vm_kmem_size_max > 0 && vm_kmem_size >= vm_kmem_size_max)
vm_kmem_size = vm_kmem_size_max;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
Allow VM_KMEM_SIZE to be tuned from the kernel environment. This tuning value *completely* overrides any value precalculated by the kernel.
1999-01-21 21:54:32 +00:00
/* Allow final override from the kernel environment */
Rename the kern.vm.kmem.size tunable to the more logical vm.kmem_size. To assure backward compatibility (conditional on !BURN_BRIDGES), look it up by its old name first, and log a warning (but accept the setting) if it was found. If both the old and new name are defined, the new name takes precedence. Also export vm.kmem_size as a read-only sysctl variable; I find it hard to tune a parameter when I don't know its default value, especially when that default value is computed at boot time.
2004-01-27 15:59:38 +00:00
#ifndef BURN_BRIDGES
if (TUNABLE_INT_FETCH("kern.vm.kmem.size", &vm_kmem_size) != 0)
printf("kern.vm.kmem.size is now called vm.kmem_size!\n");
#endif
TUNABLE_INT_FETCH("vm.kmem_size", &vm_kmem_size);
Allow VM_KMEM_SIZE to be tuned from the kernel environment. This tuning value *completely* overrides any value precalculated by the kernel.
1999-01-21 21:54:32 +00:00
Fixed sign and overflow bugs that caused the allocation size of the kernel malloc region (kmem_map) to be wrong and semi-random on systems with more than 1GB of RAM. This is not a complete fix, but is sufficient for machines with 4GB or less of memory. A complete fix will require some changes to the getenv stuff so that 64bit values can be passed around. NOT FIXED: machines with more than 4GB of RAM (e.g. some large Alphas) since we're still using ints to hold some of the values. Reviewed by: bde
2000-01-28 04:04:58 +00:00
/*
* Limit kmem virtual size to twice the physical memory.
* This allows for kmem map sparseness, but limits the size
* to something sane. Be careful to not overflow the 32bit
* ints while doing the check.
*/
Don't overflow when calculating vm_kmem_size. This fixes kmem_map too small panics on PAE machines which have odd > 4GB sizes (4.5 gig would render a 20MB of KVA for kmem_map instead of 200MB). Submitted by: John Cagle <john.cagle@hp.com>, jeff Reviewed by: jeff, peter, scottl, lots of USENIX folks
2003-06-11 05:18:59 +00:00
if (((vm_kmem_size / 2) / PAGE_SIZE) > cnt.v_page_count)
Fixed sign and overflow bugs that caused the allocation size of the kernel malloc region (kmem_map) to be wrong and semi-random on systems with more than 1GB of RAM. This is not a complete fix, but is sufficient for machines with 4GB or less of memory. A complete fix will require some changes to the getenv stuff so that 64bit values can be passed around. NOT FIXED: machines with more than 4GB of RAM (e.g. some large Alphas) since we're still using ints to hold some of the values. Reviewed by: bde
2000-01-28 04:04:58 +00:00
vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE;
Try to dynamically size the VM_KMEM_SIZE (but is still able to be overridden in a way identically as before.) I had problems with the system properly handling the number of vnodes when there is alot of system memory, and the default VM_KMEM_SIZE. Two new options "VM_KMEM_SIZE_SCALE" and "VM_KMEM_SIZE_MAX" have been added to support better auto-sizing for systems with greater than 128MB.
1998-02-23 07:41:23 +00:00
Add init_param3() to subr_param. This function is called immediately after the kernel map has been sized, and is the optimal place for the autosizing of memory allocations which occur within the kernel map to occur. Suggested by: bde
2003-07-11 00:01:03 +00:00
/*
* Tune settings based on the kernel map's size at this time.
*/
init_param3(vm_kmem_size / PAGE_SIZE);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
kmem_map = kmem_suballoc(kernel_map, (vm_offset_t *)&kmembase,
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
(vm_offset_t *)&kmemlimit, vm_kmem_size);
Get rid of the ad-hoc memory allocator for vm_map_entries, in lieu of a simple, clean zone type allocator. This new allocator will also be used for machine dependent pmap PV entries.
1997-08-05 00:02:08 +00:00
kmem_map->system_map = 1;
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
- Split UMA_ZFLAG_OFFPAGE into UMA_ZFLAG_OFFPAGE and UMA_ZFLAG_HASH. - Remove all instances of the mallochash. - Stash the slab pointer in the vm page's object pointer when allocating from the kmem_obj. - Use the overloaded object pointer to find slabs for malloced memory.
2002-09-18 08:26:30 +00:00
uma_startup2();
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
for (i = 0, indx = 0; kmemzones[indx].kz_size != 0; indx++) {
int size = kmemzones[indx].kz_size;
char *name = kmemzones[indx].kz_name;
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
Add a new UMA debugging facility. This will overwrite freed memory with 0xdeadc0de and then check for it just before memory is handed off as part of a new request. This will catch any post free/pre alloc modification of memory, as well as introduce errors for anything that tries to dereference it as a pointer. This code takes the form of special init, fini, ctor and dtor routines that are specificly used by malloc. It is in a seperate file because additional debugging aids will want to live here as well.
2002-04-30 07:54:25 +00:00
kmemzones[indx].kz_zone = uma_zcreate(name, size,
#ifdef INVARIANTS
Hide a pointer to the malloc_type bucket at the end of the freed memory. If this memory is modified after it has been freed we can now report it's previous owner.
2002-05-02 09:07:04 +00:00
mtrash_ctor, mtrash_dtor, mtrash_init, mtrash_fini,
Add a new UMA debugging facility. This will overwrite freed memory with 0xdeadc0de and then check for it just before memory is handed off as part of a new request. This will catch any post free/pre alloc modification of memory, as well as introduce errors for anything that tries to dereference it as a pointer. This code takes the form of special init, fini, ctor and dtor routines that are specificly used by malloc. It is in a seperate file because additional debugging aids will want to live here as well.
2002-04-30 07:54:25 +00:00
#else
NULL, NULL, NULL, NULL,
#endif
UMA_ALIGN_PTR, UMA_ZONE_MALLOC);
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
for (;i <= size; i+= KMEM_ZBASE)
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
kmemsize[i >> KMEM_ZSHIFT] = indx;
This is the first part of the new kernel memory allocator. This replaces malloc(9) and vm_zone with a slab like allocator. Reviewed by: arch@
2002-03-19 09:11:49 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
}
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
void
malloc_init(data)
void *data;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
{
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
struct malloc_type *type = (struct malloc_type *)data;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
mtx_lock(&malloc_mtx);
KAME netinet6 basic part(no IPsec,no V6 Multicast Forwarding, no UDP/TCP for IPv6 yet) With this patch, you can assigne IPv6 addr automatically, and can reply to IPv6 ping. Reviewed by: freebsd-arch, cvs-committers Obtained from: KAME project
1999-11-22 02:45:11 +00:00
if (type->ks_magic != M_MAGIC)
Remove the long description from the in-kernel datastructure. Put a magic field in there instead, to help catch uninitialized malloc types.
1997-10-28 19:01:02 +00:00
panic("malloc type lacks magic");
Some fixes from John Hood: 1) Fix the initialization of malloc structure that changed due to perf opt. 2) Remove unneeded include. 3) An initialization assert added to malloc. Submitted by: John Hood <cgull@smoke.marlboro.vt.us>
1997-12-05 05:36:58 +00:00
if (cnt.v_page_count == 0)
panic("malloc_init not allowed before vm init");
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
if (type->ks_next != NULL)
return;
Make malloc more extensible. The malloc type is now a pointer to the struct kmemstats that describes the type. This allows subsystems to declare their malloc types locally and <sys/malloc.h> doesn't need tweaked everytime somebody gets an idea. You can even have a type local to a lkm... I don't know if we really need the longdesc, comments welcome. TODO: There is a single nit in ext2fs, that will be fixed later, and I intend to remove all unused malloc types and distribute the rest closer to their use.
1997-10-10 14:06:34 +00:00
type->ks_next = kmemstatistics;
kmemstatistics = type;
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
mtx_init(&type->ks_mtx, type->ks_shortdesc, "Malloc Stats", MTX_DEF);
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
mtx_unlock(&malloc_mtx);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
void
malloc_uninit(data)
void *data;
{
struct malloc_type *type = (struct malloc_type *)data;
struct malloc_type *t;
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
mtx_lock(&malloc_mtx);
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
mtx_lock(&type->ks_mtx);
KAME netinet6 basic part(no IPsec,no V6 Multicast Forwarding, no UDP/TCP for IPv6 yet) With this patch, you can assigne IPv6 addr automatically, and can reply to IPv6 ping. Reviewed by: freebsd-arch, cvs-committers Obtained from: KAME project
1999-11-22 02:45:11 +00:00
if (type->ks_magic != M_MAGIC)
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
panic("malloc type lacks magic");
if (cnt.v_page_count == 0)
panic("malloc_uninit not allowed before vm init");
if (type == kmemstatistics)
kmemstatistics = type->ks_next;
else {
for (t = kmemstatistics; t->ks_next != NULL; t = t->ks_next) {
if (t->ks_next == type) {
t->ks_next = type->ks_next;
break;
}
}
}
Fixed corruption of the kmemstatistcs list. The first malloc() with malloc type at the tail of the list changed the list from linear to circular. This seemed to cause surprisingly few problems, but it now causes weird output from `vmstat -m', probably because a more important malloc type is now at the tail of the list. Fix it by abusing ks_limit instead of ks_next as a flag for being on the list. Don't forget to clear the flag when a malloc type is uninit'ed. Uninit'ing is still fundamentally broken -- it loses history.
1999-05-12 11:11:27 +00:00
type->ks_next = NULL;
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
mtx_destroy(&type->ks_mtx);
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
mtx_unlock(&malloc_mtx);
Have MALLOC_DECLARE() initialize malloc types explicitly, and have them removed at module unload (if in a module of course). However; this introduces a new dependency on <sys/kernel.h> for things that use MALLOC_DECLARE(). Bruce told me it is better to add sys/kernel.h to the handful of files that need it rather than add an extra include to sys/malloc.h for kernel compiles. Updates to follow in subsequent commits.
1998-11-10 08:46:24 +00:00
}
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
static int
sysctl_kern_malloc(SYSCTL_HANDLER_ARGS)
{
struct malloc_type *type;
int linesize = 128;
int curline;
int bufsize;
int first;
int error;
char *buf;
char *p;
int cnt;
int len;
int i;
cnt = 0;
mtx_lock(&malloc_mtx);
for (type = kmemstatistics; type != NULL; type = type->ks_next)
cnt++;
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
mtx_unlock(&malloc_mtx);
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
bufsize = linesize * (cnt + 1);
Back out M_* changes, per decision of the TRB. Approved by: trb
2003-02-19 05:47:46 +00:00
p = buf = (char *)malloc(bufsize, M_TEMP, M_WAITOK|M_ZERO);
malloc/free(9) no longer require Giant. Use the malloc_mtx to protect the mallochash. Mallochash is going to go away as soon as I introduce the kfree/kmalloc api and partially overhaul the malloc wrapper. This can't happen until all users of the malloc api that expect memory to be aligned on the size of the allocation are fixed.
2002-05-02 07:22:19 +00:00
mtx_lock(&malloc_mtx);
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
len = snprintf(p, linesize,
"\n Type InUse MemUse HighUse Requests Size(s)\n");
p += len;
for (type = kmemstatistics; cnt != 0 && type != NULL;
type = type->ks_next, cnt--) {
if (type->ks_calls == 0)
continue;
curline = linesize - 2; /* Leave room for the \n */
Convert longs to u_longs in stats. This will hold off wrap arounds for a while longer.
2002-04-30 22:39:32 +00:00
len = snprintf(p, curline, "%13s%6lu%6luK%7luK%9llu",
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
type->ks_shortdesc,
type->ks_inuse,
(type->ks_memuse + 1023) / 1024,
(type->ks_maxused + 1023) / 1024,
(long long unsigned)type->ks_calls);
curline -= len;
p += len;
first = 1;
- Replace the bandaid introduced in revision 1.110 with a better solution. - Add braces for a ``for'' statement containing a single multi-line statement.
2002-05-31 09:41:09 +00:00
for (i = 0; i < sizeof(kmemzones) / sizeof(kmemzones[0]) - 1;
i++) {
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
if (type->ks_size & (1 << i)) {
if (first)
len = snprintf(p, curline, " ");
else
len = snprintf(p, curline, ",");
curline -= len;
p += len;
len = snprintf(p, curline,
"%s", kmemzones[i].kz_name);
curline -= len;
p += len;
first = 0;
}
- Replace the bandaid introduced in revision 1.110 with a better solution. - Add braces for a ``for'' statement containing a single multi-line statement.
2002-05-31 09:41:09 +00:00
}
Remove malloc_type's ks_limit. Updated the kmemzones logic such that the ks_size bitmap can be used as an index into it to report the size of the zone used. Create the kern.malloc sysctl which replaces the kvm mechanism to report similar data. This will provide an easy place for statistics aggregation if malloc_type statistics become per cpu data. Add some code ifdef'd under MALLOC_PROFILING to facilitate a tool for sizing the malloc buckets.
2002-04-15 04:05:53 +00:00
len = snprintf(p, 2, "\n");
p += len;
}
mtx_unlock(&malloc_mtx);
error = SYSCTL_OUT(req, buf, p - buf);
free(buf, M_TEMP);
return (error);
}
SYSCTL_OID(_kern, OID_AUTO, malloc, CTLTYPE_STRING|CTLFLAG_RD,
NULL, 0, sysctl_kern_malloc, "A", "Malloc Stats");
Finish adding support code for sysctl kern.mprof. This dumps some malloc information related to bucket size effeciency. Three things are printed on each row: Size is the size the user actually asked for rounded to 16 bytes. Requests is the number of times this size was asked for. Real Size is the size we actually handed out. At the end the total memory used and total waste is displayed. Currently my system displays about 33% wasted memory. The intent of this code is to gather statistics for tuning the malloc bucket sizes. It is not intended to be run with INVARIANTS and it is not entirely mp safe. It can be enabled via 'options MALLOC_PROFILE' which was commited earlier.
2002-04-15 05:24:01 +00:00
#ifdef MALLOC_PROFILE
static int
sysctl_kern_mprof(SYSCTL_HANDLER_ARGS)
{
int linesize = 64;
uint64_t count;
uint64_t waste;
uint64_t mem;
int bufsize;
int error;
char *buf;
int rsize;
int size;
char *p;
int len;
int i;
bufsize = linesize * (KMEM_ZSIZE + 1);
bufsize += 128; /* For the stats line */
bufsize += 128; /* For the banner line */
waste = 0;
mem = 0;
Back out M_* changes, per decision of the TRB. Approved by: trb
2003-02-19 05:47:46 +00:00
p = buf = (char *)malloc(bufsize, M_TEMP, M_WAITOK|M_ZERO);
Finish adding support code for sysctl kern.mprof. This dumps some malloc information related to bucket size effeciency. Three things are printed on each row: Size is the size the user actually asked for rounded to 16 bytes. Requests is the number of times this size was asked for. Real Size is the size we actually handed out. At the end the total memory used and total waste is displayed. Currently my system displays about 33% wasted memory. The intent of this code is to gather statistics for tuning the malloc bucket sizes. It is not intended to be run with INVARIANTS and it is not entirely mp safe. It can be enabled via 'options MALLOC_PROFILE' which was commited earlier.
2002-04-15 05:24:01 +00:00
len = snprintf(p, bufsize,
"\n Size Requests Real Size\n");
bufsize -= len;
p += len;
for (i = 0; i < KMEM_ZSIZE; i++) {
size = i << KMEM_ZSHIFT;
rsize = kmemzones[kmemsize[i]].kz_size;
count = (long long unsigned)krequests[i];
len = snprintf(p, bufsize, "%6d%28llu%11d\n",
size, (unsigned long long)count, rsize);
bufsize -= len;
p += len;
if ((rsize * count) > (size * count))
waste += (rsize * count) - (size * count);
mem += (rsize * count);
}
len = snprintf(p, bufsize,
"\nTotal memory used:\t%30llu\nTotal Memory wasted:\t%30llu\n",
(unsigned long long)mem, (unsigned long long)waste);
p += len;
error = SYSCTL_OUT(req, buf, p - buf);
free(buf, M_TEMP);
return (error);
}
SYSCTL_OID(_kern, OID_AUTO, mprof, CTLTYPE_STRING|CTLFLAG_RD,
NULL, 0, sysctl_kern_mprof, "A", "Malloc Profiling");
#endif /* MALLOC_PROFILE */
Reference in New Issue Copy Permalink