2005-01-06 23:35:40 +00:00
|
|
|
/*-
|
2017-11-20 19:43:44 +00:00
|
|
|
* SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1987, 1991, 1993
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
* The Regents of the University of California.
|
2009-04-19 12:41:37 +00:00
|
|
|
* Copyright (c) 2005-2009 Robert N. M. Watson
|
2018-01-07 13:21:01 +00:00
|
|
|
* Copyright (c) 2008 Otto Moerbeek <otto@drijf.net> (mallocarray)
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
* All rights reserved.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2016-09-15 13:16:20 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
1994-05-24 10:09:53 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
|
* without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
|
*
|
|
|
|
|
* @(#)kern_malloc.c 8.3 (Berkeley) 1/4/94
|
|
|
|
|
*/
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Kernel malloc(9) implementation -- general purpose kernel memory allocator
|
|
|
|
|
* based on memory types. Back end is implemented using the UMA(9) zone
|
|
|
|
|
* allocator. A set of fixed-size buckets are used for smaller allocations,
|
|
|
|
|
* and a special UMA allocation interface is used for larger allocations.
|
|
|
|
|
* Callers declare memory types, and statistics are maintained independently
|
|
|
|
|
* for each memory type. Statistics are maintained per-CPU for performance
|
|
|
|
|
* reasons. See malloc(9) and comments in malloc.h for a detailed
|
|
|
|
|
* description.
|
|
|
|
|
*/
|
|
|
|
|
|
2003-06-11 00:56:59 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
|
2005-10-20 17:41:47 +00:00
|
|
|
#include "opt_ddb.h"
|
1998-02-23 07:41:23 +00:00
|
|
|
#include "opt_vm.h"
|
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <sys/systm.h>
|
2004-07-10 21:36:01 +00:00
|
|
|
#include <sys/kdb.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/kernel.h>
|
2001-05-01 08:13:21 +00:00
|
|
|
#include <sys/lock.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/malloc.h>
|
2000-10-20 07:29:16 +00:00
|
|
|
#include <sys/mutex.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <sys/vmmeter.h>
|
2001-01-21 19:25:07 +00:00
|
|
|
#include <sys/proc.h>
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
#include <sys/sbuf.h>
|
2002-04-15 04:05:53 +00:00
|
|
|
#include <sys/sysctl.h>
|
2002-11-01 18:58:12 +00:00
|
|
|
#include <sys/time.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <sys/vmem.h>
|
2000-09-23 00:01:37 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm.h>
|
2002-09-18 08:26:30 +00:00
|
|
|
#include <vm/pmap.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <vm/vm_pageout.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_param.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_kern.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_extern.h>
|
1997-08-05 00:02:08 +00:00
|
|
|
#include <vm/vm_map.h>
|
2002-09-18 08:26:30 +00:00
|
|
|
#include <vm/vm_page.h>
|
2002-03-19 09:11:49 +00:00
|
|
|
#include <vm/uma.h>
|
|
|
|
|
#include <vm/uma_int.h>
|
2002-04-30 07:54:25 +00:00
|
|
|
#include <vm/uma_dbg.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
|
|
|
|
#include <vm/memguard.h>
|
|
|
|
|
#endif
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
#include <vm/redzone.h>
|
|
|
|
|
#endif
|
2005-01-21 18:09:17 +00:00
|
|
|
|
1999-09-19 08:40:11 +00:00
|
|
|
#if defined(INVARIANTS) && defined(__i386__)
|
|
|
|
|
#include <machine/cpu.h>
|
|
|
|
|
#endif
|
|
|
|
|
|
2005-10-20 17:41:47 +00:00
|
|
|
#include <ddb/ddb.h>
|
|
|
|
|
|
2008-05-23 00:43:36 +00:00
|
|
|
#ifdef KDTRACE_HOOKS
|
|
|
|
|
#include <sys/dtrace_bsd.h>
|
|
|
|
|
|
2018-04-24 01:06:20 +00:00
|
|
|
bool __read_frequently dtrace_malloc_enabled;
|
|
|
|
|
dtrace_malloc_probe_func_t __read_mostly dtrace_malloc_probe;
|
2008-05-23 00:43:36 +00:00
|
|
|
#endif
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
#if defined(INVARIANTS) || defined(MALLOC_MAKE_FAILURES) || \
|
|
|
|
|
defined(DEBUG_MEMGUARD) || defined(DEBUG_REDZONE)
|
|
|
|
|
#define MALLOC_DEBUG 1
|
|
|
|
|
#endif
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/*
|
|
|
|
|
* When realloc() is called, if the new size is sufficiently smaller than
|
|
|
|
|
* the old size, realloc() will allocate a new, smaller block to avoid
|
|
|
|
|
* wasting memory. 'Sufficiently smaller' is defined as: newsize <=
|
|
|
|
|
* oldsize / 2^n, where REALLOC_FRACTION defines the value of 'n'.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef REALLOC_FRACTION
|
|
|
|
|
#define REALLOC_FRACTION 1 /* new block if <= half the size */
|
|
|
|
|
#endif
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Centrally define some common malloc types.
|
|
|
|
|
*/
|
1999-10-03 12:18:29 +00:00
|
|
|
MALLOC_DEFINE(M_CACHE, "cache", "Various Dynamically allocated caches");
|
1999-09-11 16:41:39 +00:00
|
|
|
MALLOC_DEFINE(M_DEVBUF, "devbuf", "device driver memory");
|
|
|
|
|
MALLOC_DEFINE(M_TEMP, "temp", "misc temporary data buffers");
|
|
|
|
|
|
1998-11-10 08:46:24 +00:00
|
|
|
static struct malloc_type *kmemstatistics;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
static int kmemcount;
|
2000-02-16 21:11:33 +00:00
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
#define KMEM_ZSHIFT 4
|
|
|
|
|
#define KMEM_ZBASE 16
|
|
|
|
|
#define KMEM_ZMASK (KMEM_ZBASE - 1)
|
|
|
|
|
|
2014-08-14 05:13:24 +00:00
|
|
|
#define KMEM_ZMAX 65536
|
2002-03-19 09:11:49 +00:00
|
|
|
#define KMEM_ZSIZE (KMEM_ZMAX >> KMEM_ZSHIFT)
|
2010-06-21 09:55:56 +00:00
|
|
|
static uint8_t kmemsize[KMEM_ZSIZE + 1];
|
2002-04-15 04:05:53 +00:00
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
#ifndef MALLOC_DEBUG_MAXZONES
|
|
|
|
|
#define MALLOC_DEBUG_MAXZONES 1
|
|
|
|
|
#endif
|
|
|
|
|
static int numzones = MALLOC_DEBUG_MAXZONES;
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Small malloc(9) memory allocations are allocated from a set of UMA buckets
|
|
|
|
|
* of various sizes.
|
|
|
|
|
*
|
|
|
|
|
* XXX: The comment here used to read "These won't be powers of two for
|
|
|
|
|
* long." It's possible that a significant amount of wasted memory could be
|
|
|
|
|
* recovered by tuning the sizes of these buckets.
|
|
|
|
|
*/
|
2002-03-19 09:11:49 +00:00
|
|
|
struct {
|
2002-04-15 04:05:53 +00:00
|
|
|
int kz_size;
|
|
|
|
|
char *kz_name;
|
2010-07-28 15:36:12 +00:00
|
|
|
uma_zone_t kz_zone[MALLOC_DEBUG_MAXZONES];
|
2002-04-15 04:05:53 +00:00
|
|
|
} kmemzones[] = {
|
2010-07-28 15:36:12 +00:00
|
|
|
{16, "16", },
|
|
|
|
|
{32, "32", },
|
|
|
|
|
{64, "64", },
|
|
|
|
|
{128, "128", },
|
|
|
|
|
{256, "256", },
|
|
|
|
|
{512, "512", },
|
|
|
|
|
{1024, "1024", },
|
|
|
|
|
{2048, "2048", },
|
|
|
|
|
{4096, "4096", },
|
|
|
|
|
{8192, "8192", },
|
|
|
|
|
{16384, "16384", },
|
|
|
|
|
{32768, "32768", },
|
|
|
|
|
{65536, "65536", },
|
2002-03-19 09:11:49 +00:00
|
|
|
{0, NULL},
|
1994-05-24 10:09:53 +00:00
|
|
|
};
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Zone to allocate malloc type descriptions from. For ABI reasons, memory
|
|
|
|
|
* types are described by a data structure passed by the declaring code, but
|
|
|
|
|
* the malloc(9) implementation has its own data structure describing the
|
|
|
|
|
* type and statistics. This permits the malloc(9)-internal data structures
|
|
|
|
|
* to be modified without breaking binary-compiled kernel modules that
|
|
|
|
|
* declare malloc types.
|
|
|
|
|
*/
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
static uma_zone_t mt_zone;
|
2012-11-06 04:10:32 +00:00
|
|
|
|
2008-07-05 19:34:33 +00:00
|
|
|
u_long vm_kmem_size;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_size, CTLFLAG_RDTUN, &vm_kmem_size, 0,
|
2004-01-27 15:59:38 +00:00
|
|
|
"Size of kernel memory");
|
2002-05-02 07:22:19 +00:00
|
|
|
|
2014-08-14 05:31:39 +00:00
|
|
|
static u_long kmem_zmax = KMEM_ZMAX;
|
|
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_zmax, CTLFLAG_RDTUN, &kmem_zmax, 0,
|
|
|
|
|
"Maximum allocation size that malloc(9) would use UMA as backend");
|
|
|
|
|
|
2008-07-05 19:34:33 +00:00
|
|
|
static u_long vm_kmem_size_min;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_size_min, CTLFLAG_RDTUN, &vm_kmem_size_min, 0,
|
2007-04-21 01:14:48 +00:00
|
|
|
"Minimum size of kernel memory");
|
|
|
|
|
|
2008-07-05 19:34:33 +00:00
|
|
|
static u_long vm_kmem_size_max;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_size_max, CTLFLAG_RDTUN, &vm_kmem_size_max, 0,
|
2004-09-29 14:21:40 +00:00
|
|
|
"Maximum size of kernel memory");
|
|
|
|
|
|
2014-06-28 17:36:18 +00:00
|
|
|
static u_int vm_kmem_size_scale;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_scale, CTLFLAG_RDTUN, &vm_kmem_size_scale, 0,
|
2004-09-29 14:21:40 +00:00
|
|
|
"Scale factor for kernel memory size");
|
|
|
|
|
|
2010-10-07 18:11:33 +00:00
|
|
|
static int sysctl_kmem_map_size(SYSCTL_HANDLER_ARGS);
|
|
|
|
|
SYSCTL_PROC(_vm, OID_AUTO, kmem_map_size,
|
|
|
|
|
CTLFLAG_RD | CTLTYPE_ULONG | CTLFLAG_MPSAFE, NULL, 0,
|
2013-08-07 06:21:20 +00:00
|
|
|
sysctl_kmem_map_size, "LU", "Current kmem allocation size");
|
2010-10-07 18:11:33 +00:00
|
|
|
|
2010-10-09 09:03:17 +00:00
|
|
|
static int sysctl_kmem_map_free(SYSCTL_HANDLER_ARGS);
|
|
|
|
|
SYSCTL_PROC(_vm, OID_AUTO, kmem_map_free,
|
|
|
|
|
CTLFLAG_RD | CTLTYPE_ULONG | CTLFLAG_MPSAFE, NULL, 0,
|
2013-08-07 06:21:20 +00:00
|
|
|
sysctl_kmem_map_free, "LU", "Free space in kmem");
|
2010-10-09 09:03:17 +00:00
|
|
|
|
2002-05-02 07:22:19 +00:00
|
|
|
/*
|
2002-09-18 08:26:30 +00:00
|
|
|
* The malloc_mtx protects the kmemstatistics linked list.
|
2002-05-02 07:22:19 +00:00
|
|
|
*/
|
|
|
|
|
struct mtx malloc_mtx;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-04-15 05:24:01 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
|
|
|
|
uint64_t krequests[KMEM_ZSIZE + 1];
|
|
|
|
|
|
|
|
|
|
static int sysctl_kern_mprof(SYSCTL_HANDLER_ARGS);
|
|
|
|
|
#endif
|
2002-04-15 04:05:53 +00:00
|
|
|
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
static int sysctl_kern_malloc_stats(SYSCTL_HANDLER_ARGS);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* time_uptime of the last malloc(9) failure (induced or real).
|
|
|
|
|
*/
|
2002-11-01 18:58:12 +00:00
|
|
|
static time_t t_malloc_fail;
|
|
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
#if defined(MALLOC_MAKE_FAILURES) || (MALLOC_DEBUG_MAXZONES > 1)
|
2011-11-07 15:43:11 +00:00
|
|
|
static SYSCTL_NODE(_debug, OID_AUTO, malloc, CTLFLAG_RD, 0,
|
2010-07-28 15:36:12 +00:00
|
|
|
"Kernel malloc debugging options");
|
|
|
|
|
#endif
|
|
|
|
|
|
2003-03-26 20:18:40 +00:00
|
|
|
/*
|
2006-07-23 19:51:39 +00:00
|
|
|
* malloc(9) fault injection -- cause malloc failures every (n) mallocs when
|
|
|
|
|
* the caller specifies M_NOWAIT. If set to 0, no failures are caused.
|
2003-03-26 20:18:40 +00:00
|
|
|
*/
|
2006-07-23 19:51:39 +00:00
|
|
|
#ifdef MALLOC_MAKE_FAILURES
|
2003-03-26 20:18:40 +00:00
|
|
|
static int malloc_failure_rate;
|
|
|
|
|
static int malloc_nowait_count;
|
|
|
|
|
static int malloc_failure_count;
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_debug_malloc, OID_AUTO, failure_rate, CTLFLAG_RWTUN,
|
2003-03-26 20:18:40 +00:00
|
|
|
&malloc_failure_rate, 0, "Every (n) mallocs with M_NOWAIT will fail");
|
|
|
|
|
SYSCTL_INT(_debug_malloc, OID_AUTO, failure_count, CTLFLAG_RD,
|
|
|
|
|
&malloc_failure_count, 0, "Number of imposed M_NOWAIT malloc failures");
|
|
|
|
|
#endif
|
|
|
|
|
|
2010-10-07 18:11:33 +00:00
|
|
|
static int
|
|
|
|
|
sysctl_kmem_map_size(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
|
|
|
|
u_long size;
|
|
|
|
|
|
2017-11-28 23:40:54 +00:00
|
|
|
size = uma_size();
|
2010-10-07 18:11:33 +00:00
|
|
|
return (sysctl_handle_long(oidp, &size, 0, req));
|
|
|
|
|
}
|
|
|
|
|
|
2010-10-09 09:03:17 +00:00
|
|
|
static int
|
|
|
|
|
sysctl_kmem_map_free(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
2017-11-28 23:40:54 +00:00
|
|
|
u_long size, limit;
|
2010-10-09 09:03:17 +00:00
|
|
|
|
2017-11-28 23:40:54 +00:00
|
|
|
/* The sysctl is unsigned, implement as a saturation value. */
|
|
|
|
|
size = uma_size();
|
|
|
|
|
limit = uma_limit();
|
|
|
|
|
if (size > limit)
|
|
|
|
|
size = 0;
|
|
|
|
|
else
|
|
|
|
|
size = limit - size;
|
2010-10-09 09:03:17 +00:00
|
|
|
return (sysctl_handle_long(oidp, &size, 0, req));
|
|
|
|
|
}
|
|
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
/*
|
|
|
|
|
* malloc(9) uma zone separation -- sub-page buffer overruns in one
|
|
|
|
|
* malloc type will affect only a subset of other malloc types.
|
|
|
|
|
*/
|
|
|
|
|
#if MALLOC_DEBUG_MAXZONES > 1
|
|
|
|
|
static void
|
|
|
|
|
tunable_set_numzones(void)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
TUNABLE_INT_FETCH("debug.malloc.numzones",
|
|
|
|
|
&numzones);
|
|
|
|
|
|
|
|
|
|
/* Sanity check the number of malloc uma zones. */
|
|
|
|
|
if (numzones <= 0)
|
|
|
|
|
numzones = 1;
|
|
|
|
|
if (numzones > MALLOC_DEBUG_MAXZONES)
|
|
|
|
|
numzones = MALLOC_DEBUG_MAXZONES;
|
|
|
|
|
}
|
|
|
|
|
SYSINIT(numzones, SI_SUB_TUNABLES, SI_ORDER_ANY, tunable_set_numzones, NULL);
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_debug_malloc, OID_AUTO, numzones, CTLFLAG_RDTUN | CTLFLAG_NOFETCH,
|
2010-07-28 15:36:12 +00:00
|
|
|
&numzones, 0, "Number of malloc uma subzones");
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Any number that changes regularly is an okay choice for the
|
|
|
|
|
* offset. Build numbers are pretty good of you have them.
|
|
|
|
|
*/
|
|
|
|
|
static u_int zone_offset = __FreeBSD_version;
|
|
|
|
|
TUNABLE_INT("debug.malloc.zone_offset", &zone_offset);
|
|
|
|
|
SYSCTL_UINT(_debug_malloc, OID_AUTO, zone_offset, CTLFLAG_RDTUN,
|
|
|
|
|
&zone_offset, 0, "Separate malloc types by examining the "
|
|
|
|
|
"Nth character in the malloc type short description.");
|
|
|
|
|
|
2018-04-23 22:28:49 +00:00
|
|
|
static void
|
|
|
|
|
mtp_set_subzone(struct malloc_type *mtp)
|
2010-07-28 15:36:12 +00:00
|
|
|
{
|
2018-04-23 22:28:49 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
const char *desc;
|
2010-07-28 15:36:12 +00:00
|
|
|
size_t len;
|
|
|
|
|
u_int val;
|
|
|
|
|
|
2018-04-23 22:28:49 +00:00
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
desc = mtp->ks_shortdesc;
|
2010-07-28 15:36:12 +00:00
|
|
|
if (desc == NULL || (len = strlen(desc)) == 0)
|
2018-04-23 22:28:49 +00:00
|
|
|
val = 0;
|
|
|
|
|
else
|
|
|
|
|
val = desc[zone_offset % len];
|
|
|
|
|
mtip->mti_zone = (val % numzones);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline u_int
|
|
|
|
|
mtp_get_subzone(struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
|
|
|
|
|
KASSERT(mtip->mti_zone < numzones,
|
|
|
|
|
("mti_zone %u out of range %d",
|
|
|
|
|
mtip->mti_zone, numzones));
|
|
|
|
|
return (mtip->mti_zone);
|
2010-07-28 15:36:12 +00:00
|
|
|
}
|
|
|
|
|
#elif MALLOC_DEBUG_MAXZONES == 0
|
|
|
|
|
#error "MALLOC_DEBUG_MAXZONES must be positive."
|
|
|
|
|
#else
|
2018-04-23 22:28:49 +00:00
|
|
|
static void
|
|
|
|
|
mtp_set_subzone(struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
mtip->mti_zone = 0;
|
|
|
|
|
}
|
|
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
static inline u_int
|
2018-04-23 22:28:49 +00:00
|
|
|
mtp_get_subzone(struct malloc_type *mtp)
|
2010-07-28 15:36:12 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
#endif /* MALLOC_DEBUG_MAXZONES > 1 */
|
|
|
|
|
|
2002-11-01 18:58:12 +00:00
|
|
|
int
|
|
|
|
|
malloc_last_fail(void)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (time_uptime - t_malloc_fail);
|
|
|
|
|
}
|
|
|
|
|
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
/*
|
2006-07-23 19:51:39 +00:00
|
|
|
* An allocation has succeeded -- update malloc type statistics for the
|
|
|
|
|
* amount of bucket size. Occurs within a critical section so that the
|
|
|
|
|
* thread isn't preempted and doesn't migrate while updating per-PCU
|
|
|
|
|
* statistics.
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
*/
|
|
|
|
|
static void
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_zone_allocated(struct malloc_type *mtp, unsigned long size,
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
int zindx)
|
|
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type_stats *mtsp;
|
|
|
|
|
|
|
|
|
|
critical_enter();
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
mtsp = &mtip->mti_stats[curcpu];
|
2005-07-27 23:17:31 +00:00
|
|
|
if (size > 0) {
|
|
|
|
|
mtsp->mts_memalloced += size;
|
|
|
|
|
mtsp->mts_numallocs++;
|
|
|
|
|
}
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
if (zindx != -1)
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtsp->mts_size |= 1 << zindx;
|
2008-05-23 00:43:36 +00:00
|
|
|
|
|
|
|
|
#ifdef KDTRACE_HOOKS
|
2018-04-24 01:06:20 +00:00
|
|
|
if (__predict_false(dtrace_malloc_enabled)) {
|
2008-05-23 00:43:36 +00:00
|
|
|
uint32_t probe_id = mtip->mti_probes[DTMALLOC_PROBE_MALLOC];
|
|
|
|
|
if (probe_id != 0)
|
|
|
|
|
(dtrace_malloc_probe)(probe_id,
|
|
|
|
|
(uintptr_t) mtp, (uintptr_t) mtip,
|
|
|
|
|
(uintptr_t) mtsp, size, zindx);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
critical_exit();
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_allocated(struct malloc_type *mtp, unsigned long size)
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
|
2005-07-27 23:17:31 +00:00
|
|
|
if (size > 0)
|
|
|
|
|
malloc_type_zone_allocated(mtp, size, -1);
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
2007-06-14 03:02:33 +00:00
|
|
|
* A free operation has occurred -- update malloc type statistics for the
|
2006-07-23 19:51:39 +00:00
|
|
|
* amount of the bucket size. Occurs within a critical section so that the
|
|
|
|
|
* thread isn't preempted and doesn't migrate while updating per-CPU
|
|
|
|
|
* statistics.
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
*/
|
|
|
|
|
void
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_freed(struct malloc_type *mtp, unsigned long size)
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type_stats *mtsp;
|
|
|
|
|
|
|
|
|
|
critical_enter();
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
mtsp = &mtip->mti_stats[curcpu];
|
|
|
|
|
mtsp->mts_memfreed += size;
|
|
|
|
|
mtsp->mts_numfrees++;
|
2008-05-23 00:43:36 +00:00
|
|
|
|
|
|
|
|
#ifdef KDTRACE_HOOKS
|
2018-04-24 01:06:20 +00:00
|
|
|
if (__predict_false(dtrace_malloc_enabled)) {
|
2008-05-23 00:43:36 +00:00
|
|
|
uint32_t probe_id = mtip->mti_probes[DTMALLOC_PROBE_FREE];
|
|
|
|
|
if (probe_id != 0)
|
|
|
|
|
(dtrace_malloc_probe)(probe_id,
|
|
|
|
|
(uintptr_t) mtp, (uintptr_t) mtip,
|
|
|
|
|
(uintptr_t) mtsp, size, 0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
critical_exit();
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
}
|
|
|
|
|
|
2011-10-27 02:52:24 +00:00
|
|
|
/*
|
|
|
|
|
* contigmalloc:
|
|
|
|
|
*
|
|
|
|
|
* Allocate a block of physically contiguous memory.
|
|
|
|
|
*
|
|
|
|
|
* If M_NOWAIT is set, this routine will not block and return NULL if
|
|
|
|
|
* the allocation fails.
|
|
|
|
|
*/
|
|
|
|
|
void *
|
|
|
|
|
contigmalloc(unsigned long size, struct malloc_type *type, int flags,
|
|
|
|
|
vm_paddr_t low, vm_paddr_t high, unsigned long alignment,
|
2012-03-01 19:58:34 +00:00
|
|
|
vm_paddr_t boundary)
|
2011-10-27 02:52:24 +00:00
|
|
|
{
|
|
|
|
|
void *ret;
|
|
|
|
|
|
2013-08-07 06:21:20 +00:00
|
|
|
ret = (void *)kmem_alloc_contig(kernel_arena, size, flags, low, high,
|
2011-10-27 02:52:24 +00:00
|
|
|
alignment, boundary, VM_MEMATTR_DEFAULT);
|
|
|
|
|
if (ret != NULL)
|
|
|
|
|
malloc_type_allocated(type, round_page(size));
|
|
|
|
|
return (ret);
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
void *
|
|
|
|
|
contigmalloc_domain(unsigned long size, struct malloc_type *type,
|
|
|
|
|
int domain, int flags, vm_paddr_t low, vm_paddr_t high,
|
|
|
|
|
unsigned long alignment, vm_paddr_t boundary)
|
|
|
|
|
{
|
|
|
|
|
void *ret;
|
|
|
|
|
|
|
|
|
|
ret = (void *)kmem_alloc_contig_domain(domain, size, flags, low, high,
|
|
|
|
|
alignment, boundary, VM_MEMATTR_DEFAULT);
|
|
|
|
|
if (ret != NULL)
|
|
|
|
|
malloc_type_allocated(type, round_page(size));
|
|
|
|
|
return (ret);
|
|
|
|
|
}
|
|
|
|
|
|
2011-10-27 02:52:24 +00:00
|
|
|
/*
|
|
|
|
|
* contigfree:
|
|
|
|
|
*
|
|
|
|
|
* Free a block of memory allocated by contigmalloc.
|
|
|
|
|
*
|
|
|
|
|
* This routine may not block.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
contigfree(void *addr, unsigned long size, struct malloc_type *type)
|
|
|
|
|
{
|
|
|
|
|
|
2013-08-07 06:21:20 +00:00
|
|
|
kmem_free(kernel_arena, (vm_offset_t)addr, size);
|
2011-10-27 02:52:24 +00:00
|
|
|
malloc_type_freed(type, round_page(size));
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
static int
|
2018-01-25 06:37:14 +00:00
|
|
|
malloc_dbg(caddr_t *vap, size_t *sizep, struct malloc_type *mtp,
|
2018-01-12 23:25:05 +00:00
|
|
|
int flags)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef INVARIANTS
|
2002-04-15 04:05:53 +00:00
|
|
|
int indx;
|
1997-10-10 14:06:34 +00:00
|
|
|
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC, ("malloc: bad malloc type magic"));
|
2003-03-10 19:39:53 +00:00
|
|
|
/*
|
2005-08-02 20:03:23 +00:00
|
|
|
* Check that exactly one of M_WAITOK or M_NOWAIT is specified.
|
2003-03-10 19:39:53 +00:00
|
|
|
*/
|
2005-08-02 20:03:23 +00:00
|
|
|
indx = flags & (M_WAITOK | M_NOWAIT);
|
2003-03-10 19:39:53 +00:00
|
|
|
if (indx != M_NOWAIT && indx != M_WAITOK) {
|
|
|
|
|
static struct timeval lasterr;
|
|
|
|
|
static int curerr, once;
|
|
|
|
|
if (once == 0 && ppsratecheck(&lasterr, &curerr, 1)) {
|
|
|
|
|
printf("Bad malloc flags: %x\n", indx);
|
2004-07-10 21:36:01 +00:00
|
|
|
kdb_backtrace();
|
2003-03-10 19:39:53 +00:00
|
|
|
flags |= M_WAITOK;
|
|
|
|
|
once++;
|
|
|
|
|
}
|
|
|
|
|
}
|
2003-03-10 20:24:54 +00:00
|
|
|
#endif
|
2003-03-26 20:18:40 +00:00
|
|
|
#ifdef MALLOC_MAKE_FAILURES
|
|
|
|
|
if ((flags & M_NOWAIT) && (malloc_failure_rate != 0)) {
|
|
|
|
|
atomic_add_int(&malloc_nowait_count, 1);
|
|
|
|
|
if ((malloc_nowait_count % malloc_failure_rate) == 0) {
|
|
|
|
|
atomic_add_int(&malloc_failure_count, 1);
|
2003-04-25 21:49:24 +00:00
|
|
|
t_malloc_fail = time_uptime;
|
2018-01-12 23:25:05 +00:00
|
|
|
*vap = NULL;
|
|
|
|
|
return (EJUSTRETURN);
|
2003-03-26 20:18:40 +00:00
|
|
|
}
|
|
|
|
|
}
|
2002-04-23 18:50:25 +00:00
|
|
|
#endif
|
2018-05-10 17:55:24 +00:00
|
|
|
if (flags & M_WAITOK) {
|
2001-09-12 08:38:13 +00:00
|
|
|
KASSERT(curthread->td_intr_nesting_level == 0,
|
2003-02-19 05:47:46 +00:00
|
|
|
("malloc(M_WAITOK) in interrupt context"));
|
2018-05-10 17:55:24 +00:00
|
|
|
KASSERT(curthread->td_epochnest == 0,
|
|
|
|
|
("malloc(M_WAITOK) in epoch context"));
|
|
|
|
|
}
|
2015-12-11 20:05:07 +00:00
|
|
|
KASSERT(curthread->td_critnest == 0 || SCHEDULER_STOPPED(),
|
2015-11-19 14:04:53 +00:00
|
|
|
("malloc: called with spinlock or critical section held"));
|
|
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2018-01-12 23:25:05 +00:00
|
|
|
if (memguard_cmp_mtp(mtp, *sizep)) {
|
|
|
|
|
*vap = memguard_alloc(*sizep, flags);
|
|
|
|
|
if (*vap != NULL)
|
|
|
|
|
return (EJUSTRETURN);
|
2010-08-11 22:10:37 +00:00
|
|
|
/* This is unfortunate but should not be fatal. */
|
|
|
|
|
}
|
2005-01-21 18:09:17 +00:00
|
|
|
#endif
|
|
|
|
|
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
2018-01-12 23:25:05 +00:00
|
|
|
*sizep = redzone_size_ntor(*sizep);
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* malloc:
|
|
|
|
|
*
|
|
|
|
|
* Allocate a block of memory.
|
|
|
|
|
*
|
|
|
|
|
* If M_NOWAIT is set, this routine will not block and return NULL if
|
|
|
|
|
* the allocation fails.
|
|
|
|
|
*/
|
|
|
|
|
void *
|
2018-06-02 22:20:09 +00:00
|
|
|
(malloc)(size_t size, struct malloc_type *mtp, int flags)
|
2018-01-12 23:25:05 +00:00
|
|
|
{
|
|
|
|
|
int indx;
|
|
|
|
|
caddr_t va;
|
|
|
|
|
uma_zone_t zone;
|
|
|
|
|
#if defined(DEBUG_REDZONE)
|
|
|
|
|
unsigned long osize = size;
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
2018-05-19 04:43:49 +00:00
|
|
|
va = NULL;
|
2018-01-12 23:25:05 +00:00
|
|
|
if (malloc_dbg(&va, &size, mtp, flags) != 0)
|
|
|
|
|
return (va);
|
2006-01-31 11:09:21 +00:00
|
|
|
#endif
|
|
|
|
|
|
2014-08-14 05:31:39 +00:00
|
|
|
if (size <= kmem_zmax) {
|
2002-04-15 04:05:53 +00:00
|
|
|
if (size & KMEM_ZMASK)
|
|
|
|
|
size = (size & ~KMEM_ZMASK) + KMEM_ZBASE;
|
|
|
|
|
indx = kmemsize[size >> KMEM_ZSHIFT];
|
2018-04-23 22:28:49 +00:00
|
|
|
zone = kmemzones[indx].kz_zone[mtp_get_subzone(mtp)];
|
2002-04-15 04:05:53 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
|
|
|
|
krequests[size >> KMEM_ZSHIFT]++;
|
|
|
|
|
#endif
|
2002-03-19 09:11:49 +00:00
|
|
|
va = uma_zalloc(zone, flags);
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
if (va != NULL)
|
2009-01-25 09:11:24 +00:00
|
|
|
size = zone->uz_size;
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_zone_allocated(mtp, va == NULL ? 0 : size, indx);
|
2002-03-19 09:11:49 +00:00
|
|
|
} else {
|
2002-04-15 04:05:53 +00:00
|
|
|
size = roundup(size, PAGE_SIZE);
|
2002-03-19 09:11:49 +00:00
|
|
|
zone = NULL;
|
|
|
|
|
va = uma_large_malloc(size, flags);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_allocated(mtp, va == NULL ? 0 : size);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2003-05-12 05:09:56 +00:00
|
|
|
if (flags & M_WAITOK)
|
2003-02-19 05:47:46 +00:00
|
|
|
KASSERT(va != NULL, ("malloc(M_WAITOK) returned NULL"));
|
2003-05-12 05:09:56 +00:00
|
|
|
else if (va == NULL)
|
2002-11-01 18:58:12 +00:00
|
|
|
t_malloc_fail = time_uptime;
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
if (va != NULL)
|
|
|
|
|
va = redzone_setup(va, osize);
|
|
|
|
|
#endif
|
|
|
|
|
return ((void *) va);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void *
|
2018-01-24 19:37:18 +00:00
|
|
|
malloc_domain(size_t size, struct malloc_type *mtp, int domain,
|
2018-01-12 23:25:05 +00:00
|
|
|
int flags)
|
|
|
|
|
{
|
|
|
|
|
int indx;
|
|
|
|
|
caddr_t va;
|
|
|
|
|
uma_zone_t zone;
|
|
|
|
|
#if defined(DEBUG_REDZONE)
|
|
|
|
|
unsigned long osize = size;
|
2006-01-31 11:09:21 +00:00
|
|
|
#endif
|
2018-01-12 23:25:05 +00:00
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
2018-05-19 04:43:49 +00:00
|
|
|
va = NULL;
|
2018-01-12 23:25:05 +00:00
|
|
|
if (malloc_dbg(&va, &size, mtp, flags) != 0)
|
|
|
|
|
return (va);
|
|
|
|
|
#endif
|
|
|
|
|
if (size <= kmem_zmax) {
|
|
|
|
|
if (size & KMEM_ZMASK)
|
|
|
|
|
size = (size & ~KMEM_ZMASK) + KMEM_ZBASE;
|
|
|
|
|
indx = kmemsize[size >> KMEM_ZSHIFT];
|
2018-04-23 22:28:49 +00:00
|
|
|
zone = kmemzones[indx].kz_zone[mtp_get_subzone(mtp)];
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
|
|
|
|
krequests[size >> KMEM_ZSHIFT]++;
|
|
|
|
|
#endif
|
|
|
|
|
va = uma_zalloc_domain(zone, NULL, domain, flags);
|
|
|
|
|
if (va != NULL)
|
|
|
|
|
size = zone->uz_size;
|
|
|
|
|
malloc_type_zone_allocated(mtp, va == NULL ? 0 : size, indx);
|
|
|
|
|
} else {
|
|
|
|
|
size = roundup(size, PAGE_SIZE);
|
|
|
|
|
zone = NULL;
|
|
|
|
|
va = uma_large_malloc_domain(size, domain, flags);
|
|
|
|
|
malloc_type_allocated(mtp, va == NULL ? 0 : size);
|
|
|
|
|
}
|
|
|
|
|
if (flags & M_WAITOK)
|
|
|
|
|
KASSERT(va != NULL, ("malloc(M_WAITOK) returned NULL"));
|
|
|
|
|
else if (va == NULL)
|
|
|
|
|
t_malloc_fail = time_uptime;
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
if (va != NULL)
|
|
|
|
|
va = redzone_setup(va, osize);
|
2003-02-01 10:07:49 +00:00
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
return ((void *) va);
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-07 13:21:01 +00:00
|
|
|
void *
|
|
|
|
|
mallocarray(size_t nmemb, size_t size, struct malloc_type *type, int flags)
|
|
|
|
|
{
|
|
|
|
|
|
2018-01-10 21:49:45 +00:00
|
|
|
if (WOULD_OVERFLOW(nmemb, size))
|
|
|
|
|
panic("mallocarray: %zu * %zu overflowed", nmemb, size);
|
2018-01-07 13:21:01 +00:00
|
|
|
|
|
|
|
|
return (malloc(size * nmemb, type, flags));
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef INVARIANTS
|
|
|
|
|
static void
|
|
|
|
|
free_save_type(void *addr, struct malloc_type *mtp, u_long size)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2018-01-12 23:25:05 +00:00
|
|
|
struct malloc_type **mtpp = addr;
|
1997-10-10 14:06:34 +00:00
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
/*
|
|
|
|
|
* Cache a pointer to the malloc_type that most recently freed
|
|
|
|
|
* this memory here. This way we know who is most likely to
|
|
|
|
|
* have stepped on it later.
|
|
|
|
|
*
|
|
|
|
|
* This code assumes that size is a multiple of 8 bytes for
|
|
|
|
|
* 64 bit machines
|
|
|
|
|
*/
|
|
|
|
|
mtpp = (struct malloc_type **) ((unsigned long)mtpp & ~UMA_ALIGN_PTR);
|
|
|
|
|
mtpp += (size - sizeof(struct malloc_type *)) /
|
|
|
|
|
sizeof(struct malloc_type *);
|
|
|
|
|
*mtpp = mtp;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
static int
|
|
|
|
|
free_dbg(void **addrp, struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
void *addr;
|
|
|
|
|
|
|
|
|
|
addr = *addrp;
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC, ("free: bad malloc type magic"));
|
2015-12-11 20:05:07 +00:00
|
|
|
KASSERT(curthread->td_critnest == 0 || SCHEDULER_STOPPED(),
|
2015-11-19 14:04:53 +00:00
|
|
|
("free: called with spinlock or critical section held"));
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/* free(NULL, ...) does nothing */
|
|
|
|
|
if (addr == NULL)
|
2018-01-12 23:25:05 +00:00
|
|
|
return (EJUSTRETURN);
|
2002-03-13 01:42:33 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2010-08-11 22:10:37 +00:00
|
|
|
if (is_memguard_addr(addr)) {
|
2005-01-21 18:09:17 +00:00
|
|
|
memguard_free(addr);
|
2018-01-12 23:25:05 +00:00
|
|
|
return (EJUSTRETURN);
|
2005-01-21 18:09:17 +00:00
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
redzone_check(addr);
|
2018-01-12 23:25:05 +00:00
|
|
|
*addrp = redzone_addr_ntor(addr);
|
2006-01-31 11:09:21 +00:00
|
|
|
#endif
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
/*
|
|
|
|
|
* free:
|
|
|
|
|
*
|
|
|
|
|
* Free a block of memory allocated by malloc.
|
|
|
|
|
*
|
|
|
|
|
* This routine may not block.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
free(void *addr, struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
uma_slab_t slab;
|
|
|
|
|
u_long size;
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
if (free_dbg(&addr, mtp) != 0)
|
|
|
|
|
return;
|
|
|
|
|
#endif
|
|
|
|
|
/* free(NULL, ...) does nothing */
|
|
|
|
|
if (addr == NULL)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
slab = vtoslab((vm_offset_t)addr & (~UMA_SLAB_MASK));
|
2002-03-19 09:11:49 +00:00
|
|
|
if (slab == NULL)
|
2002-04-15 04:05:53 +00:00
|
|
|
panic("free: address %p(%p) has not been allocated.\n",
|
2002-09-18 08:26:30 +00:00
|
|
|
addr, (void *)((u_long)addr & (~UMA_SLAB_MASK)));
|
|
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
if (!(slab->us_flags & UMA_SLAB_MALLOC)) {
|
2018-01-12 23:25:05 +00:00
|
|
|
size = slab->us_keg->uk_size;
|
2002-05-02 09:07:04 +00:00
|
|
|
#ifdef INVARIANTS
|
2018-01-12 23:25:05 +00:00
|
|
|
free_save_type(addr, mtp, size);
|
|
|
|
|
#endif
|
|
|
|
|
uma_zfree_arg(LIST_FIRST(&slab->us_keg->uk_zones), addr, slab);
|
|
|
|
|
} else {
|
|
|
|
|
size = slab->us_size;
|
|
|
|
|
uma_large_free(slab);
|
|
|
|
|
}
|
|
|
|
|
malloc_type_freed(mtp, size);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
free_domain(void *addr, struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
uma_slab_t slab;
|
|
|
|
|
u_long size;
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
if (free_dbg(&addr, mtp) != 0)
|
|
|
|
|
return;
|
2002-05-02 09:07:04 +00:00
|
|
|
#endif
|
2018-01-12 23:25:05 +00:00
|
|
|
|
|
|
|
|
/* free(NULL, ...) does nothing */
|
|
|
|
|
if (addr == NULL)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
slab = vtoslab((vm_offset_t)addr & (~UMA_SLAB_MASK));
|
|
|
|
|
if (slab == NULL)
|
|
|
|
|
panic("free_domain: address %p(%p) has not been allocated.\n",
|
|
|
|
|
addr, (void *)((u_long)addr & (~UMA_SLAB_MASK)));
|
|
|
|
|
|
|
|
|
|
if (!(slab->us_flags & UMA_SLAB_MALLOC)) {
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
size = slab->us_keg->uk_size;
|
2002-05-02 09:07:04 +00:00
|
|
|
#ifdef INVARIANTS
|
2018-01-12 23:25:05 +00:00
|
|
|
free_save_type(addr, mtp, size);
|
2002-05-02 09:07:04 +00:00
|
|
|
#endif
|
2018-01-12 23:25:05 +00:00
|
|
|
uma_zfree_domain(LIST_FIRST(&slab->us_keg->uk_zones),
|
|
|
|
|
addr, slab);
|
2002-03-19 09:11:49 +00:00
|
|
|
} else {
|
|
|
|
|
size = slab->us_size;
|
|
|
|
|
uma_large_free(slab);
|
1999-05-06 18:13:11 +00:00
|
|
|
}
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_freed(mtp, size);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/*
|
|
|
|
|
* realloc: change the size of a memory block
|
|
|
|
|
*/
|
|
|
|
|
void *
|
2018-01-24 19:37:18 +00:00
|
|
|
realloc(void *addr, size_t size, struct malloc_type *mtp, int flags)
|
2002-03-13 01:42:33 +00:00
|
|
|
{
|
2002-03-19 09:11:49 +00:00
|
|
|
uma_slab_t slab;
|
2002-03-13 01:42:33 +00:00
|
|
|
unsigned long alloc;
|
|
|
|
|
void *newaddr;
|
|
|
|
|
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC,
|
|
|
|
|
("realloc: bad malloc type magic"));
|
2015-12-11 20:05:07 +00:00
|
|
|
KASSERT(curthread->td_critnest == 0 || SCHEDULER_STOPPED(),
|
2015-11-19 14:04:53 +00:00
|
|
|
("realloc: called with spinlock or critical section held"));
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/* realloc(NULL, ...) is equivalent to malloc(...) */
|
|
|
|
|
if (addr == NULL)
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
return (malloc(size, mtp, flags));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* XXX: Should report free of old memory and alloc of new memory to
|
|
|
|
|
* per-CPU stats.
|
|
|
|
|
*/
|
2002-03-13 01:42:33 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2010-08-31 16:57:58 +00:00
|
|
|
if (is_memguard_addr(addr))
|
|
|
|
|
return (memguard_realloc(addr, size, mtp, flags));
|
2005-01-21 18:09:17 +00:00
|
|
|
#endif
|
|
|
|
|
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
slab = NULL;
|
|
|
|
|
alloc = redzone_get_size(addr);
|
|
|
|
|
#else
|
2002-09-18 08:26:30 +00:00
|
|
|
slab = vtoslab((vm_offset_t)addr & ~(UMA_SLAB_MASK));
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/* Sanity check */
|
2002-03-19 09:11:49 +00:00
|
|
|
KASSERT(slab != NULL,
|
2002-03-13 01:42:33 +00:00
|
|
|
("realloc: address %p out of range", (void *)addr));
|
|
|
|
|
|
|
|
|
|
/* Get the size of the original block */
|
2005-12-28 01:53:13 +00:00
|
|
|
if (!(slab->us_flags & UMA_SLAB_MALLOC))
|
Bring in mbuma to replace mballoc.
mbuma is an Mbuf & Cluster allocator built on top of a number of
extensions to the UMA framework, all included herein.
Extensions to UMA worth noting:
- Better layering between slab <-> zone caches; introduce
Keg structure which splits off slab cache away from the
zone structure and allows multiple zones to be stacked
on top of a single Keg (single type of slab cache);
perhaps we should look into defining a subset API on
top of the Keg for special use by malloc(9),
for example.
- UMA_ZONE_REFCNT zones can now be added, and reference
counters automagically allocated for them within the end
of the associated slab structures. uma_find_refcnt()
does a kextract to fetch the slab struct reference from
the underlying page, and lookup the corresponding refcnt.
mbuma things worth noting:
- integrates mbuf & cluster allocations with extended UMA
and provides caches for commonly-allocated items; defines
several zones (two primary, one secondary) and two kegs.
- change up certain code paths that always used to do:
m_get() + m_clget() to instead just use m_getcl() and
try to take advantage of the newly defined secondary
Packet zone.
- netstat(1) and systat(1) quickly hacked up to do basic
stat reporting but additional stats work needs to be
done once some other details within UMA have been taken
care of and it becomes clearer to how stats will work
within the modified framework.
From the user perspective, one implication is that the
NMBCLUSTERS compile-time option is no longer used. The
maximum number of clusters is still capped off according
to maxusers, but it can be made unlimited by setting
the kern.ipc.nmbclusters boot-time tunable to zero.
Work should be done to write an appropriate sysctl
handler allowing dynamic tuning of kern.ipc.nmbclusters
at runtime.
Additional things worth noting/known issues (READ):
- One report of 'ips' (ServeRAID) driver acting really
slow in conjunction with mbuma. Need more data.
Latest report is that ips is equally sucking with
and without mbuma.
- Giant leak in NFS code sometimes occurs, can't
reproduce but currently analyzing; brueffer is
able to reproduce but THIS IS NOT an mbuma-specific
problem and currently occurs even WITHOUT mbuma.
- Issues in network locking: there is at least one
code path in the rip code where one or more locks
are acquired and we end up in m_prepend() with
M_WAITOK, which causes WITNESS to whine from within
UMA. Current temporary solution: force all UMA
allocations to be M_NOWAIT from within UMA for now
to avoid deadlocks unless WITNESS is defined and we
can determine with certainty that we're not holding
any locks when we're M_WAITOK.
- I've seen at least one weird socketbuffer empty-but-
mbuf-still-attached panic. I don't believe this
to be related to mbuma but please keep your eyes
open, turn on debugging, and capture crash dumps.
This change removes more code than it adds.
A paper is available detailing the change and considering
various performance issues, it was presented at BSDCan2004:
http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf
Please read the paper for Future Work and implementation
details, as well as credits.
Testing and Debugging:
rwatson,
brueffer,
Ketrien I. Saihr-Kesenchedra,
...
Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
|
|
|
alloc = slab->us_keg->uk_size;
|
2002-03-19 09:11:49 +00:00
|
|
|
else
|
|
|
|
|
alloc = slab->us_size;
|
2002-03-13 01:42:33 +00:00
|
|
|
|
|
|
|
|
/* Reuse the original block if appropriate */
|
|
|
|
|
if (size <= alloc
|
|
|
|
|
&& (size > (alloc >> REALLOC_FRACTION) || alloc == MINALLOCSIZE))
|
|
|
|
|
return (addr);
|
2006-01-31 11:09:21 +00:00
|
|
|
#endif /* !DEBUG_REDZONE */
|
2002-03-13 01:42:33 +00:00
|
|
|
|
|
|
|
|
/* Allocate a new, bigger (or smaller) block */
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
if ((newaddr = malloc(size, mtp, flags)) == NULL)
|
2002-03-13 01:42:33 +00:00
|
|
|
return (NULL);
|
|
|
|
|
|
|
|
|
|
/* Copy over original contents */
|
|
|
|
|
bcopy(addr, newaddr, min(size, alloc));
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
free(addr, mtp);
|
2002-03-13 01:42:33 +00:00
|
|
|
return (newaddr);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* reallocf: same as realloc() but free memory on failure.
|
|
|
|
|
*/
|
|
|
|
|
void *
|
2018-01-24 19:37:18 +00:00
|
|
|
reallocf(void *addr, size_t size, struct malloc_type *mtp, int flags)
|
2002-03-13 01:42:33 +00:00
|
|
|
{
|
|
|
|
|
void *mem;
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
if ((mem = realloc(addr, size, mtp, flags)) == NULL)
|
|
|
|
|
free(addr, mtp);
|
2002-03-13 01:42:33 +00:00
|
|
|
return (mem);
|
|
|
|
|
}
|
|
|
|
|
|
2014-02-23 17:37:24 +00:00
|
|
|
#ifndef __sparc64__
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
CTASSERT(VM_KMEM_SIZE_SCALE >= 1);
|
2014-02-23 17:37:24 +00:00
|
|
|
#endif
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
|
2013-08-07 06:21:20 +00:00
|
|
|
/*
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* Initialize the kernel memory (kmem) arena.
|
2013-08-07 06:21:20 +00:00
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
kmeminit(void)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2014-06-28 03:56:17 +00:00
|
|
|
u_long mem_size;
|
|
|
|
|
u_long tmp;
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
|
2014-06-28 03:56:17 +00:00
|
|
|
#ifdef VM_KMEM_SIZE
|
|
|
|
|
if (vm_kmem_size == 0)
|
|
|
|
|
vm_kmem_size = VM_KMEM_SIZE;
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef VM_KMEM_SIZE_MIN
|
|
|
|
|
if (vm_kmem_size_min == 0)
|
|
|
|
|
vm_kmem_size_min = VM_KMEM_SIZE_MIN;
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef VM_KMEM_SIZE_MAX
|
|
|
|
|
if (vm_kmem_size_max == 0)
|
|
|
|
|
vm_kmem_size_max = VM_KMEM_SIZE_MAX;
|
|
|
|
|
#endif
|
1998-02-23 07:41:23 +00:00
|
|
|
/*
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* Calculate the amount of kernel virtual address (KVA) space that is
|
|
|
|
|
* preallocated to the kmem arena. In order to support a wide range
|
|
|
|
|
* of machines, it is a function of the physical memory size,
|
|
|
|
|
* specifically,
|
|
|
|
|
*
|
|
|
|
|
* min(max(physical memory size / VM_KMEM_SIZE_SCALE,
|
|
|
|
|
* VM_KMEM_SIZE_MIN), VM_KMEM_SIZE_MAX)
|
1998-02-23 07:41:23 +00:00
|
|
|
*
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* Every architecture must define an integral value for
|
|
|
|
|
* VM_KMEM_SIZE_SCALE. However, the definitions of VM_KMEM_SIZE_MIN
|
|
|
|
|
* and VM_KMEM_SIZE_MAX, which represent respectively the floor and
|
|
|
|
|
* ceiling on this preallocation, are optional. Typically,
|
|
|
|
|
* VM_KMEM_SIZE_MAX is itself a function of the available KVA space on
|
|
|
|
|
* a given architecture.
|
1998-02-23 07:41:23 +00:00
|
|
|
*/
|
2014-03-22 10:26:09 +00:00
|
|
|
mem_size = vm_cnt.v_page_count;
|
2014-09-22 05:07:22 +00:00
|
|
|
if (mem_size <= 32768) /* delphij XXX 128MB */
|
|
|
|
|
kmem_zmax = PAGE_SIZE;
|
1998-02-23 07:41:23 +00:00
|
|
|
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
if (vm_kmem_size_scale < 1)
|
|
|
|
|
vm_kmem_size_scale = VM_KMEM_SIZE_SCALE;
|
|
|
|
|
|
2014-06-28 03:56:17 +00:00
|
|
|
/*
|
|
|
|
|
* Check if we should use defaults for the "vm_kmem_size"
|
|
|
|
|
* variable:
|
|
|
|
|
*/
|
|
|
|
|
if (vm_kmem_size == 0) {
|
|
|
|
|
vm_kmem_size = (mem_size / vm_kmem_size_scale) * PAGE_SIZE;
|
2014-06-27 22:05:21 +00:00
|
|
|
|
2014-06-28 03:56:17 +00:00
|
|
|
if (vm_kmem_size_min > 0 && vm_kmem_size < vm_kmem_size_min)
|
|
|
|
|
vm_kmem_size = vm_kmem_size_min;
|
|
|
|
|
if (vm_kmem_size_max > 0 && vm_kmem_size >= vm_kmem_size_max)
|
|
|
|
|
vm_kmem_size = vm_kmem_size_max;
|
|
|
|
|
}
|
1998-02-23 07:41:23 +00:00
|
|
|
|
2000-01-28 04:04:58 +00:00
|
|
|
/*
|
2014-06-28 03:56:17 +00:00
|
|
|
* The amount of KVA space that is preallocated to the
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* kmem arena can be set statically at compile-time or manually
|
|
|
|
|
* through the kernel environment. However, it is still limited to
|
|
|
|
|
* twice the physical memory size, which has been sufficient to handle
|
|
|
|
|
* the most severe cases of external fragmentation in the kmem arena.
|
2000-01-28 04:04:58 +00:00
|
|
|
*/
|
2011-12-07 07:03:14 +00:00
|
|
|
if (vm_kmem_size / 2 / PAGE_SIZE > mem_size)
|
|
|
|
|
vm_kmem_size = 2 * mem_size * PAGE_SIZE;
|
1998-02-23 07:41:23 +00:00
|
|
|
|
2013-08-09 22:30:54 +00:00
|
|
|
vm_kmem_size = round_page(vm_kmem_size);
|
2010-08-11 22:10:37 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2012-07-15 20:29:48 +00:00
|
|
|
tmp = memguard_fudge(vm_kmem_size, kernel_map);
|
2010-08-11 22:10:37 +00:00
|
|
|
#else
|
|
|
|
|
tmp = vm_kmem_size;
|
|
|
|
|
#endif
|
2017-11-28 23:40:54 +00:00
|
|
|
uma_set_limit(tmp);
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
|
|
|
|
/*
|
|
|
|
|
* Initialize MemGuard if support compiled in. MemGuard is a
|
|
|
|
|
* replacement allocator used for detecting tamper-after-free
|
|
|
|
|
* scenarios as they occur. It is only used for debugging.
|
|
|
|
|
*/
|
2017-11-28 23:40:54 +00:00
|
|
|
memguard_init(kernel_arena);
|
2005-01-21 18:09:17 +00:00
|
|
|
#endif
|
2013-08-07 06:21:20 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initialize the kernel memory allocator
|
|
|
|
|
*/
|
|
|
|
|
/* ARGSUSED*/
|
|
|
|
|
static void
|
|
|
|
|
mallocinit(void *dummy)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
uint8_t indx;
|
|
|
|
|
|
|
|
|
|
mtx_init(&malloc_mtx, "malloc", NULL, MTX_DEF);
|
|
|
|
|
|
|
|
|
|
kmeminit();
|
2005-01-21 18:09:17 +00:00
|
|
|
|
2014-08-14 05:31:39 +00:00
|
|
|
if (kmem_zmax < PAGE_SIZE || kmem_zmax > KMEM_ZMAX)
|
|
|
|
|
kmem_zmax = KMEM_ZMAX;
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mt_zone = uma_zcreate("mt_zone", sizeof(struct malloc_type_internal),
|
|
|
|
|
#ifdef INVARIANTS
|
|
|
|
|
mtrash_ctor, mtrash_dtor, mtrash_init, mtrash_fini,
|
|
|
|
|
#else
|
|
|
|
|
NULL, NULL, NULL, NULL,
|
|
|
|
|
#endif
|
|
|
|
|
UMA_ALIGN_PTR, UMA_ZONE_MALLOC);
|
2002-04-15 04:05:53 +00:00
|
|
|
for (i = 0, indx = 0; kmemzones[indx].kz_size != 0; indx++) {
|
|
|
|
|
int size = kmemzones[indx].kz_size;
|
|
|
|
|
char *name = kmemzones[indx].kz_name;
|
2010-07-28 15:36:12 +00:00
|
|
|
int subzone;
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
for (subzone = 0; subzone < numzones; subzone++) {
|
|
|
|
|
kmemzones[indx].kz_zone[subzone] =
|
|
|
|
|
uma_zcreate(name, size,
|
2002-04-30 07:54:25 +00:00
|
|
|
#ifdef INVARIANTS
|
2010-07-28 15:36:12 +00:00
|
|
|
mtrash_ctor, mtrash_dtor, mtrash_init, mtrash_fini,
|
2002-04-30 07:54:25 +00:00
|
|
|
#else
|
2010-07-28 15:36:12 +00:00
|
|
|
NULL, NULL, NULL, NULL,
|
2002-04-30 07:54:25 +00:00
|
|
|
#endif
|
2010-07-28 15:36:12 +00:00
|
|
|
UMA_ALIGN_PTR, UMA_ZONE_MALLOC);
|
|
|
|
|
}
|
2002-03-19 09:11:49 +00:00
|
|
|
for (;i <= size; i+= KMEM_ZBASE)
|
2002-04-15 04:05:53 +00:00
|
|
|
kmemsize[i >> KMEM_ZSHIFT] = indx;
|
2014-08-14 05:31:39 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1997-10-10 14:06:34 +00:00
|
|
|
}
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSINIT(kmem, SI_SUB_KMEM, SI_ORDER_SECOND, mallocinit, NULL);
|
1997-10-10 14:06:34 +00:00
|
|
|
|
1998-11-10 08:46:24 +00:00
|
|
|
void
|
2005-04-12 23:54:34 +00:00
|
|
|
malloc_init(void *data)
|
1997-10-10 14:06:34 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type *mtp;
|
1997-10-10 14:06:34 +00:00
|
|
|
|
2014-03-22 10:26:09 +00:00
|
|
|
KASSERT(vm_cnt.v_page_count != 0, ("malloc_register before vm_init"));
|
1997-10-28 19:01:02 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtp = data;
|
2009-06-05 09:16:52 +00:00
|
|
|
if (mtp->ks_magic != M_MAGIC)
|
|
|
|
|
panic("malloc_init: bad malloc type magic");
|
2009-04-19 12:41:37 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtip = uma_zalloc(mt_zone, M_WAITOK | M_ZERO);
|
|
|
|
|
mtp->ks_handle = mtip;
|
2018-04-23 22:28:49 +00:00
|
|
|
mtp_set_subzone(mtp);
|
1997-12-05 05:36:58 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
mtp->ks_next = kmemstatistics;
|
|
|
|
|
kmemstatistics = mtp;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
kmemcount++;
|
2002-04-15 04:05:53 +00:00
|
|
|
mtx_unlock(&malloc_mtx);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1998-11-10 08:46:24 +00:00
|
|
|
|
|
|
|
|
void
|
2005-04-12 23:54:34 +00:00
|
|
|
malloc_uninit(void *data)
|
1998-11-10 08:46:24 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
2005-11-03 13:48:59 +00:00
|
|
|
struct malloc_type_stats *mtsp;
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type *mtp, *temp;
|
2006-03-03 22:36:52 +00:00
|
|
|
uma_slab_t slab;
|
2005-11-03 13:48:59 +00:00
|
|
|
long temp_allocs, temp_bytes;
|
|
|
|
|
int i;
|
1998-11-10 08:46:24 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtp = data;
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC,
|
|
|
|
|
("malloc_uninit: bad malloc type magic"));
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
KASSERT(mtp->ks_handle != NULL, ("malloc_deregister: cookie NULL"));
|
2009-04-19 12:41:37 +00:00
|
|
|
|
2002-04-15 04:05:53 +00:00
|
|
|
mtx_lock(&malloc_mtx);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
mtp->ks_handle = NULL;
|
|
|
|
|
if (mtp != kmemstatistics) {
|
|
|
|
|
for (temp = kmemstatistics; temp != NULL;
|
|
|
|
|
temp = temp->ks_next) {
|
2009-06-05 09:16:52 +00:00
|
|
|
if (temp->ks_next == mtp) {
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
temp->ks_next = mtp->ks_next;
|
2009-06-05 09:16:52 +00:00
|
|
|
break;
|
|
|
|
|
}
|
1998-11-10 08:46:24 +00:00
|
|
|
}
|
2009-06-05 09:16:52 +00:00
|
|
|
KASSERT(temp,
|
|
|
|
|
("malloc_uninit: type '%s' not found", mtp->ks_shortdesc));
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
} else
|
|
|
|
|
kmemstatistics = mtp->ks_next;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
kmemcount--;
|
2002-04-15 04:05:53 +00:00
|
|
|
mtx_unlock(&malloc_mtx);
|
2005-11-03 13:48:59 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Look for memory leaks.
|
|
|
|
|
*/
|
|
|
|
|
temp_allocs = temp_bytes = 0;
|
|
|
|
|
for (i = 0; i < MAXCPU; i++) {
|
|
|
|
|
mtsp = &mtip->mti_stats[i];
|
|
|
|
|
temp_allocs += mtsp->mts_numallocs;
|
|
|
|
|
temp_allocs -= mtsp->mts_numfrees;
|
|
|
|
|
temp_bytes += mtsp->mts_memalloced;
|
|
|
|
|
temp_bytes -= mtsp->mts_memfreed;
|
|
|
|
|
}
|
|
|
|
|
if (temp_allocs > 0 || temp_bytes > 0) {
|
|
|
|
|
printf("Warning: memory type %s leaked memory on destroy "
|
|
|
|
|
"(%ld allocations, %ld bytes leaked).\n", mtp->ks_shortdesc,
|
|
|
|
|
temp_allocs, temp_bytes);
|
|
|
|
|
}
|
|
|
|
|
|
2006-03-03 22:36:52 +00:00
|
|
|
slab = vtoslab((vm_offset_t) mtip & (~UMA_SLAB_MASK));
|
|
|
|
|
uma_zfree_arg(mt_zone, mtip, slab);
|
1998-11-10 08:46:24 +00:00
|
|
|
}
|
2002-04-15 04:05:53 +00:00
|
|
|
|
2005-12-30 11:45:07 +00:00
|
|
|
struct malloc_type *
|
|
|
|
|
malloc_desc2type(const char *desc)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type *mtp;
|
|
|
|
|
|
|
|
|
|
mtx_assert(&malloc_mtx, MA_OWNED);
|
|
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
if (strcmp(mtp->ks_shortdesc, desc) == 0)
|
|
|
|
|
return (mtp);
|
|
|
|
|
}
|
|
|
|
|
return (NULL);
|
|
|
|
|
}
|
|
|
|
|
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
static int
|
|
|
|
|
sysctl_kern_malloc_stats(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_stream_header mtsh;
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type_header mth;
|
|
|
|
|
struct malloc_type *mtp;
|
2010-09-16 16:13:12 +00:00
|
|
|
int error, i;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
struct sbuf sbuf;
|
|
|
|
|
|
2011-01-27 00:34:12 +00:00
|
|
|
error = sysctl_wire_old_buffer(req, 0);
|
|
|
|
|
if (error != 0)
|
|
|
|
|
return (error);
|
2010-09-16 16:13:12 +00:00
|
|
|
sbuf_new_for_sysctl(&sbuf, NULL, 128, req);
|
2015-03-14 17:08:28 +00:00
|
|
|
sbuf_clear_flags(&sbuf, SBUF_INCLUDENUL);
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert stream header.
|
|
|
|
|
*/
|
|
|
|
|
bzero(&mtsh, sizeof(mtsh));
|
|
|
|
|
mtsh.mtsh_version = MALLOC_TYPE_STREAM_VERSION;
|
|
|
|
|
mtsh.mtsh_maxcpus = MAXCPU;
|
|
|
|
|
mtsh.mtsh_count = kmemcount;
|
2010-09-16 16:13:12 +00:00
|
|
|
(void)sbuf_bcat(&sbuf, &mtsh, sizeof(mtsh));
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert alternating sequence of type headers and type statistics.
|
|
|
|
|
*/
|
|
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
mtip = (struct malloc_type_internal *)mtp->ks_handle;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert type header.
|
|
|
|
|
*/
|
|
|
|
|
bzero(&mth, sizeof(mth));
|
|
|
|
|
strlcpy(mth.mth_name, mtp->ks_shortdesc, MALLOC_MAX_NAME);
|
2010-09-16 16:13:12 +00:00
|
|
|
(void)sbuf_bcat(&sbuf, &mth, sizeof(mth));
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert type statistics for each CPU.
|
|
|
|
|
*/
|
|
|
|
|
for (i = 0; i < MAXCPU; i++) {
|
2010-09-16 16:13:12 +00:00
|
|
|
(void)sbuf_bcat(&sbuf, &mtip->mti_stats[i],
|
|
|
|
|
sizeof(mtip->mti_stats[i]));
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
mtx_unlock(&malloc_mtx);
|
2010-09-16 16:13:12 +00:00
|
|
|
error = sbuf_finish(&sbuf);
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
sbuf_delete(&sbuf);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
SYSCTL_PROC(_kern, OID_AUTO, malloc_stats, CTLFLAG_RD|CTLTYPE_STRUCT,
|
|
|
|
|
0, 0, sysctl_kern_malloc_stats, "s,malloc_type_ustats",
|
|
|
|
|
"Return malloc types");
|
|
|
|
|
|
|
|
|
|
SYSCTL_INT(_kern, OID_AUTO, malloc_count, CTLFLAG_RD, &kmemcount, 0,
|
|
|
|
|
"Count of kernel malloc types");
|
|
|
|
|
|
2008-05-23 00:43:36 +00:00
|
|
|
void
|
|
|
|
|
malloc_type_list(malloc_type_list_func_t *func, void *arg)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type *mtp, **bufmtp;
|
|
|
|
|
int count, i;
|
|
|
|
|
size_t buflen;
|
|
|
|
|
|
|
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
restart:
|
|
|
|
|
mtx_assert(&malloc_mtx, MA_OWNED);
|
|
|
|
|
count = kmemcount;
|
|
|
|
|
mtx_unlock(&malloc_mtx);
|
|
|
|
|
|
|
|
|
|
buflen = sizeof(struct malloc_type *) * count;
|
|
|
|
|
bufmtp = malloc(buflen, M_TEMP, M_WAITOK);
|
|
|
|
|
|
|
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
|
|
|
|
|
if (count < kmemcount) {
|
|
|
|
|
free(bufmtp, M_TEMP);
|
|
|
|
|
goto restart;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
for (mtp = kmemstatistics, i = 0; mtp != NULL; mtp = mtp->ks_next, i++)
|
|
|
|
|
bufmtp[i] = mtp;
|
|
|
|
|
|
|
|
|
|
mtx_unlock(&malloc_mtx);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < count; i++)
|
|
|
|
|
(func)(bufmtp[i], arg);
|
|
|
|
|
|
|
|
|
|
free(bufmtp, M_TEMP);
|
|
|
|
|
}
|
|
|
|
|
|
2005-10-20 17:41:47 +00:00
|
|
|
#ifdef DDB
|
|
|
|
|
DB_SHOW_COMMAND(malloc, db_show_malloc)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type *mtp;
|
2010-06-21 09:55:56 +00:00
|
|
|
uint64_t allocs, frees;
|
|
|
|
|
uint64_t alloced, freed;
|
2005-10-20 17:41:47 +00:00
|
|
|
int i;
|
|
|
|
|
|
2006-10-26 10:17:13 +00:00
|
|
|
db_printf("%18s %12s %12s %12s\n", "Type", "InUse", "MemUse",
|
|
|
|
|
"Requests");
|
2005-10-20 17:41:47 +00:00
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
mtip = (struct malloc_type_internal *)mtp->ks_handle;
|
|
|
|
|
allocs = 0;
|
|
|
|
|
frees = 0;
|
2006-10-26 10:17:13 +00:00
|
|
|
alloced = 0;
|
|
|
|
|
freed = 0;
|
2005-10-20 17:41:47 +00:00
|
|
|
for (i = 0; i < MAXCPU; i++) {
|
|
|
|
|
allocs += mtip->mti_stats[i].mts_numallocs;
|
|
|
|
|
frees += mtip->mti_stats[i].mts_numfrees;
|
2006-10-26 10:17:13 +00:00
|
|
|
alloced += mtip->mti_stats[i].mts_memalloced;
|
|
|
|
|
freed += mtip->mti_stats[i].mts_memfreed;
|
2005-10-20 17:41:47 +00:00
|
|
|
}
|
2006-10-26 10:17:13 +00:00
|
|
|
db_printf("%18s %12ju %12juK %12ju\n",
|
|
|
|
|
mtp->ks_shortdesc, allocs - frees,
|
|
|
|
|
(alloced - freed + 1023) / 1024, allocs);
|
2012-07-02 16:14:52 +00:00
|
|
|
if (db_pager_quit)
|
|
|
|
|
break;
|
2005-10-20 17:41:47 +00:00
|
|
|
}
|
|
|
|
|
}
|
2010-07-28 15:36:12 +00:00
|
|
|
|
|
|
|
|
#if MALLOC_DEBUG_MAXZONES > 1
|
|
|
|
|
DB_SHOW_COMMAND(multizone_matches, db_show_multizone_matches)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type *mtp;
|
|
|
|
|
u_int subzone;
|
|
|
|
|
|
|
|
|
|
if (!have_addr) {
|
|
|
|
|
db_printf("Usage: show multizone_matches <malloc type/addr>\n");
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
mtp = (void *)addr;
|
|
|
|
|
if (mtp->ks_magic != M_MAGIC) {
|
|
|
|
|
db_printf("Magic %lx does not match expected %x\n",
|
|
|
|
|
mtp->ks_magic, M_MAGIC);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
subzone = mtip->mti_zone;
|
|
|
|
|
|
|
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
if (mtip->mti_zone != subzone)
|
|
|
|
|
continue;
|
|
|
|
|
db_printf("%s\n", mtp->ks_shortdesc);
|
2012-07-02 16:14:52 +00:00
|
|
|
if (db_pager_quit)
|
|
|
|
|
break;
|
2010-07-28 15:36:12 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
#endif /* MALLOC_DEBUG_MAXZONES > 1 */
|
|
|
|
|
#endif /* DDB */
|
2005-10-20 17:41:47 +00:00
|
|
|
|
2002-04-15 05:24:01 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
|
|
|
|
|
|
|
|
|
static int
|
|
|
|
|
sysctl_kern_mprof(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct sbuf sbuf;
|
2002-04-15 05:24:01 +00:00
|
|
|
uint64_t count;
|
|
|
|
|
uint64_t waste;
|
|
|
|
|
uint64_t mem;
|
|
|
|
|
int error;
|
|
|
|
|
int rsize;
|
|
|
|
|
int size;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
waste = 0;
|
|
|
|
|
mem = 0;
|
|
|
|
|
|
2011-01-27 00:34:12 +00:00
|
|
|
error = sysctl_wire_old_buffer(req, 0);
|
|
|
|
|
if (error != 0)
|
|
|
|
|
return (error);
|
2010-09-16 16:13:12 +00:00
|
|
|
sbuf_new_for_sysctl(&sbuf, NULL, 128, req);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_printf(&sbuf,
|
2002-04-15 05:24:01 +00:00
|
|
|
"\n Size Requests Real Size\n");
|
|
|
|
|
for (i = 0; i < KMEM_ZSIZE; i++) {
|
|
|
|
|
size = i << KMEM_ZSHIFT;
|
|
|
|
|
rsize = kmemzones[kmemsize[i]].kz_size;
|
|
|
|
|
count = (long long unsigned)krequests[i];
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_printf(&sbuf, "%6d%28llu%11d\n", size,
|
|
|
|
|
(unsigned long long)count, rsize);
|
2002-04-15 05:24:01 +00:00
|
|
|
|
|
|
|
|
if ((rsize * count) > (size * count))
|
|
|
|
|
waste += (rsize * count) - (size * count);
|
|
|
|
|
mem += (rsize * count);
|
|
|
|
|
}
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_printf(&sbuf,
|
2002-04-15 05:24:01 +00:00
|
|
|
"\nTotal memory used:\t%30llu\nTotal Memory wasted:\t%30llu\n",
|
|
|
|
|
(unsigned long long)mem, (unsigned long long)waste);
|
2010-09-16 16:13:12 +00:00
|
|
|
error = sbuf_finish(&sbuf);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_delete(&sbuf);
|
2002-04-15 05:24:01 +00:00
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
SYSCTL_OID(_kern, OID_AUTO, mprof, CTLTYPE_STRING|CTLFLAG_RD,
|
|
|
|
|
NULL, 0, sysctl_kern_mprof, "A", "Malloc Profiling");
|
|
|
|
|
#endif /* MALLOC_PROFILE */
|
