2005-01-06 23:35:40 +00:00
|
|
|
/*-
|
2017-11-20 19:43:44 +00:00
|
|
|
* SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1987, 1991, 1993
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
* The Regents of the University of California.
|
2009-04-19 12:41:37 +00:00
|
|
|
* Copyright (c) 2005-2009 Robert N. M. Watson
|
2018-01-07 13:21:01 +00:00
|
|
|
* Copyright (c) 2008 Otto Moerbeek <otto@drijf.net> (mallocarray)
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
* All rights reserved.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2016-09-15 13:16:20 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
1994-05-24 10:09:53 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
|
* without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
|
*
|
|
|
|
|
* @(#)kern_malloc.c 8.3 (Berkeley) 1/4/94
|
|
|
|
|
*/
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Kernel malloc(9) implementation -- general purpose kernel memory allocator
|
|
|
|
|
* based on memory types. Back end is implemented using the UMA(9) zone
|
|
|
|
|
* allocator. A set of fixed-size buckets are used for smaller allocations,
|
|
|
|
|
* and a special UMA allocation interface is used for larger allocations.
|
|
|
|
|
* Callers declare memory types, and statistics are maintained independently
|
|
|
|
|
* for each memory type. Statistics are maintained per-CPU for performance
|
|
|
|
|
* reasons. See malloc(9) and comments in malloc.h for a detailed
|
|
|
|
|
* description.
|
|
|
|
|
*/
|
|
|
|
|
|
2003-06-11 00:56:59 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
|
2005-10-20 17:41:47 +00:00
|
|
|
#include "opt_ddb.h"
|
1998-02-23 07:41:23 +00:00
|
|
|
#include "opt_vm.h"
|
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <sys/systm.h>
|
2004-07-10 21:36:01 +00:00
|
|
|
#include <sys/kdb.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/kernel.h>
|
2001-05-01 08:13:21 +00:00
|
|
|
#include <sys/lock.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/malloc.h>
|
2000-10-20 07:29:16 +00:00
|
|
|
#include <sys/mutex.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <sys/vmmeter.h>
|
2001-01-21 19:25:07 +00:00
|
|
|
#include <sys/proc.h>
|
2019-11-29 03:14:10 +00:00
|
|
|
#include <sys/queue.h>
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
#include <sys/sbuf.h>
|
2018-09-23 19:00:06 +00:00
|
|
|
#include <sys/smp.h>
|
2002-04-15 04:05:53 +00:00
|
|
|
#include <sys/sysctl.h>
|
2002-11-01 18:58:12 +00:00
|
|
|
#include <sys/time.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <sys/vmem.h>
|
2019-10-15 21:32:38 +00:00
|
|
|
#ifdef EPOCH_TRACE
|
|
|
|
|
#include <sys/epoch.h>
|
|
|
|
|
#endif
|
2000-09-23 00:01:37 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm.h>
|
2002-09-18 08:26:30 +00:00
|
|
|
#include <vm/pmap.h>
|
2018-10-30 18:26:34 +00:00
|
|
|
#include <vm/vm_domainset.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <vm/vm_pageout.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_param.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_kern.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_extern.h>
|
1997-08-05 00:02:08 +00:00
|
|
|
#include <vm/vm_map.h>
|
2002-09-18 08:26:30 +00:00
|
|
|
#include <vm/vm_page.h>
|
2019-11-29 03:14:10 +00:00
|
|
|
#include <vm/vm_phys.h>
|
|
|
|
|
#include <vm/vm_pagequeue.h>
|
2002-03-19 09:11:49 +00:00
|
|
|
#include <vm/uma.h>
|
|
|
|
|
#include <vm/uma_int.h>
|
2002-04-30 07:54:25 +00:00
|
|
|
#include <vm/uma_dbg.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
|
|
|
|
#include <vm/memguard.h>
|
|
|
|
|
#endif
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
#include <vm/redzone.h>
|
|
|
|
|
#endif
|
2005-01-21 18:09:17 +00:00
|
|
|
|
1999-09-19 08:40:11 +00:00
|
|
|
#if defined(INVARIANTS) && defined(__i386__)
|
|
|
|
|
#include <machine/cpu.h>
|
|
|
|
|
#endif
|
|
|
|
|
|
2005-10-20 17:41:47 +00:00
|
|
|
#include <ddb/ddb.h>
|
|
|
|
|
|
2008-05-23 00:43:36 +00:00
|
|
|
#ifdef KDTRACE_HOOKS
|
|
|
|
|
#include <sys/dtrace_bsd.h>
|
|
|
|
|
|
2018-04-24 01:06:20 +00:00
|
|
|
bool __read_frequently dtrace_malloc_enabled;
|
|
|
|
|
dtrace_malloc_probe_func_t __read_mostly dtrace_malloc_probe;
|
2008-05-23 00:43:36 +00:00
|
|
|
#endif
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
#if defined(INVARIANTS) || defined(MALLOC_MAKE_FAILURES) || \
|
|
|
|
|
defined(DEBUG_MEMGUARD) || defined(DEBUG_REDZONE)
|
|
|
|
|
#define MALLOC_DEBUG 1
|
|
|
|
|
#endif
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/*
|
|
|
|
|
* When realloc() is called, if the new size is sufficiently smaller than
|
|
|
|
|
* the old size, realloc() will allocate a new, smaller block to avoid
|
|
|
|
|
* wasting memory. 'Sufficiently smaller' is defined as: newsize <=
|
|
|
|
|
* oldsize / 2^n, where REALLOC_FRACTION defines the value of 'n'.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef REALLOC_FRACTION
|
|
|
|
|
#define REALLOC_FRACTION 1 /* new block if <= half the size */
|
|
|
|
|
#endif
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Centrally define some common malloc types.
|
|
|
|
|
*/
|
1999-10-03 12:18:29 +00:00
|
|
|
MALLOC_DEFINE(M_CACHE, "cache", "Various Dynamically allocated caches");
|
1999-09-11 16:41:39 +00:00
|
|
|
MALLOC_DEFINE(M_DEVBUF, "devbuf", "device driver memory");
|
|
|
|
|
MALLOC_DEFINE(M_TEMP, "temp", "misc temporary data buffers");
|
|
|
|
|
|
1998-11-10 08:46:24 +00:00
|
|
|
static struct malloc_type *kmemstatistics;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
static int kmemcount;
|
2000-02-16 21:11:33 +00:00
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
#define KMEM_ZSHIFT 4
|
|
|
|
|
#define KMEM_ZBASE 16
|
|
|
|
|
#define KMEM_ZMASK (KMEM_ZBASE - 1)
|
|
|
|
|
|
2014-08-14 05:13:24 +00:00
|
|
|
#define KMEM_ZMAX 65536
|
2002-03-19 09:11:49 +00:00
|
|
|
#define KMEM_ZSIZE (KMEM_ZMAX >> KMEM_ZSHIFT)
|
2010-06-21 09:55:56 +00:00
|
|
|
static uint8_t kmemsize[KMEM_ZSIZE + 1];
|
2002-04-15 04:05:53 +00:00
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
#ifndef MALLOC_DEBUG_MAXZONES
|
|
|
|
|
#define MALLOC_DEBUG_MAXZONES 1
|
|
|
|
|
#endif
|
|
|
|
|
static int numzones = MALLOC_DEBUG_MAXZONES;
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Small malloc(9) memory allocations are allocated from a set of UMA buckets
|
|
|
|
|
* of various sizes.
|
|
|
|
|
*
|
|
|
|
|
* XXX: The comment here used to read "These won't be powers of two for
|
|
|
|
|
* long." It's possible that a significant amount of wasted memory could be
|
|
|
|
|
* recovered by tuning the sizes of these buckets.
|
|
|
|
|
*/
|
2002-03-19 09:11:49 +00:00
|
|
|
struct {
|
2002-04-15 04:05:53 +00:00
|
|
|
int kz_size;
|
2020-02-23 03:32:04 +00:00
|
|
|
const char *kz_name;
|
2010-07-28 15:36:12 +00:00
|
|
|
uma_zone_t kz_zone[MALLOC_DEBUG_MAXZONES];
|
2002-04-15 04:05:53 +00:00
|
|
|
} kmemzones[] = {
|
2010-07-28 15:36:12 +00:00
|
|
|
{16, "16", },
|
|
|
|
|
{32, "32", },
|
|
|
|
|
{64, "64", },
|
|
|
|
|
{128, "128", },
|
|
|
|
|
{256, "256", },
|
|
|
|
|
{512, "512", },
|
|
|
|
|
{1024, "1024", },
|
|
|
|
|
{2048, "2048", },
|
|
|
|
|
{4096, "4096", },
|
|
|
|
|
{8192, "8192", },
|
|
|
|
|
{16384, "16384", },
|
|
|
|
|
{32768, "32768", },
|
|
|
|
|
{65536, "65536", },
|
2002-03-19 09:11:49 +00:00
|
|
|
{0, NULL},
|
1994-05-24 10:09:53 +00:00
|
|
|
};
|
|
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* Zone to allocate malloc type descriptions from. For ABI reasons, memory
|
|
|
|
|
* types are described by a data structure passed by the declaring code, but
|
|
|
|
|
* the malloc(9) implementation has its own data structure describing the
|
|
|
|
|
* type and statistics. This permits the malloc(9)-internal data structures
|
|
|
|
|
* to be modified without breaking binary-compiled kernel modules that
|
|
|
|
|
* declare malloc types.
|
|
|
|
|
*/
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
static uma_zone_t mt_zone;
|
2018-09-23 19:00:06 +00:00
|
|
|
static uma_zone_t mt_stats_zone;
|
2012-11-06 04:10:32 +00:00
|
|
|
|
2008-07-05 19:34:33 +00:00
|
|
|
u_long vm_kmem_size;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_size, CTLFLAG_RDTUN, &vm_kmem_size, 0,
|
2004-01-27 15:59:38 +00:00
|
|
|
"Size of kernel memory");
|
2002-05-02 07:22:19 +00:00
|
|
|
|
2014-08-14 05:31:39 +00:00
|
|
|
static u_long kmem_zmax = KMEM_ZMAX;
|
|
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_zmax, CTLFLAG_RDTUN, &kmem_zmax, 0,
|
|
|
|
|
"Maximum allocation size that malloc(9) would use UMA as backend");
|
|
|
|
|
|
2008-07-05 19:34:33 +00:00
|
|
|
static u_long vm_kmem_size_min;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_size_min, CTLFLAG_RDTUN, &vm_kmem_size_min, 0,
|
2007-04-21 01:14:48 +00:00
|
|
|
"Minimum size of kernel memory");
|
|
|
|
|
|
2008-07-05 19:34:33 +00:00
|
|
|
static u_long vm_kmem_size_max;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_ULONG(_vm, OID_AUTO, kmem_size_max, CTLFLAG_RDTUN, &vm_kmem_size_max, 0,
|
2004-09-29 14:21:40 +00:00
|
|
|
"Maximum size of kernel memory");
|
|
|
|
|
|
2014-06-28 17:36:18 +00:00
|
|
|
static u_int vm_kmem_size_scale;
|
2010-09-30 16:45:27 +00:00
|
|
|
SYSCTL_UINT(_vm, OID_AUTO, kmem_size_scale, CTLFLAG_RDTUN, &vm_kmem_size_scale, 0,
|
2004-09-29 14:21:40 +00:00
|
|
|
"Scale factor for kernel memory size");
|
|
|
|
|
|
2010-10-07 18:11:33 +00:00
|
|
|
static int sysctl_kmem_map_size(SYSCTL_HANDLER_ARGS);
|
|
|
|
|
SYSCTL_PROC(_vm, OID_AUTO, kmem_map_size,
|
|
|
|
|
CTLFLAG_RD | CTLTYPE_ULONG | CTLFLAG_MPSAFE, NULL, 0,
|
2013-08-07 06:21:20 +00:00
|
|
|
sysctl_kmem_map_size, "LU", "Current kmem allocation size");
|
2010-10-07 18:11:33 +00:00
|
|
|
|
2010-10-09 09:03:17 +00:00
|
|
|
static int sysctl_kmem_map_free(SYSCTL_HANDLER_ARGS);
|
|
|
|
|
SYSCTL_PROC(_vm, OID_AUTO, kmem_map_free,
|
|
|
|
|
CTLFLAG_RD | CTLTYPE_ULONG | CTLFLAG_MPSAFE, NULL, 0,
|
2013-08-07 06:21:20 +00:00
|
|
|
sysctl_kmem_map_free, "LU", "Free space in kmem");
|
2010-10-09 09:03:17 +00:00
|
|
|
|
2002-05-02 07:22:19 +00:00
|
|
|
/*
|
2002-09-18 08:26:30 +00:00
|
|
|
* The malloc_mtx protects the kmemstatistics linked list.
|
2002-05-02 07:22:19 +00:00
|
|
|
*/
|
|
|
|
|
struct mtx malloc_mtx;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-04-15 05:24:01 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
|
|
|
|
uint64_t krequests[KMEM_ZSIZE + 1];
|
|
|
|
|
|
|
|
|
|
static int sysctl_kern_mprof(SYSCTL_HANDLER_ARGS);
|
|
|
|
|
#endif
|
2002-04-15 04:05:53 +00:00
|
|
|
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
static int sysctl_kern_malloc_stats(SYSCTL_HANDLER_ARGS);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2006-07-23 19:51:39 +00:00
|
|
|
/*
|
|
|
|
|
* time_uptime of the last malloc(9) failure (induced or real).
|
|
|
|
|
*/
|
2002-11-01 18:58:12 +00:00
|
|
|
static time_t t_malloc_fail;
|
|
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
#if defined(MALLOC_MAKE_FAILURES) || (MALLOC_DEBUG_MAXZONES > 1)
|
2020-02-26 14:26:36 +00:00
|
|
|
static SYSCTL_NODE(_debug, OID_AUTO, malloc, CTLFLAG_RD | CTLFLAG_MPSAFE, 0,
|
2010-07-28 15:36:12 +00:00
|
|
|
"Kernel malloc debugging options");
|
|
|
|
|
#endif
|
|
|
|
|
|
2003-03-26 20:18:40 +00:00
|
|
|
/*
|
2006-07-23 19:51:39 +00:00
|
|
|
* malloc(9) fault injection -- cause malloc failures every (n) mallocs when
|
|
|
|
|
* the caller specifies M_NOWAIT. If set to 0, no failures are caused.
|
2003-03-26 20:18:40 +00:00
|
|
|
*/
|
2006-07-23 19:51:39 +00:00
|
|
|
#ifdef MALLOC_MAKE_FAILURES
|
2003-03-26 20:18:40 +00:00
|
|
|
static int malloc_failure_rate;
|
|
|
|
|
static int malloc_nowait_count;
|
|
|
|
|
static int malloc_failure_count;
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_debug_malloc, OID_AUTO, failure_rate, CTLFLAG_RWTUN,
|
2003-03-26 20:18:40 +00:00
|
|
|
&malloc_failure_rate, 0, "Every (n) mallocs with M_NOWAIT will fail");
|
|
|
|
|
SYSCTL_INT(_debug_malloc, OID_AUTO, failure_count, CTLFLAG_RD,
|
|
|
|
|
&malloc_failure_count, 0, "Number of imposed M_NOWAIT malloc failures");
|
|
|
|
|
#endif
|
|
|
|
|
|
2010-10-07 18:11:33 +00:00
|
|
|
static int
|
|
|
|
|
sysctl_kmem_map_size(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
|
|
|
|
u_long size;
|
|
|
|
|
|
2017-11-28 23:40:54 +00:00
|
|
|
size = uma_size();
|
2010-10-07 18:11:33 +00:00
|
|
|
return (sysctl_handle_long(oidp, &size, 0, req));
|
|
|
|
|
}
|
|
|
|
|
|
2010-10-09 09:03:17 +00:00
|
|
|
static int
|
|
|
|
|
sysctl_kmem_map_free(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
2017-11-28 23:40:54 +00:00
|
|
|
u_long size, limit;
|
2010-10-09 09:03:17 +00:00
|
|
|
|
2017-11-28 23:40:54 +00:00
|
|
|
/* The sysctl is unsigned, implement as a saturation value. */
|
|
|
|
|
size = uma_size();
|
|
|
|
|
limit = uma_limit();
|
|
|
|
|
if (size > limit)
|
|
|
|
|
size = 0;
|
|
|
|
|
else
|
|
|
|
|
size = limit - size;
|
2010-10-09 09:03:17 +00:00
|
|
|
return (sysctl_handle_long(oidp, &size, 0, req));
|
|
|
|
|
}
|
|
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
/*
|
|
|
|
|
* malloc(9) uma zone separation -- sub-page buffer overruns in one
|
|
|
|
|
* malloc type will affect only a subset of other malloc types.
|
|
|
|
|
*/
|
|
|
|
|
#if MALLOC_DEBUG_MAXZONES > 1
|
|
|
|
|
static void
|
|
|
|
|
tunable_set_numzones(void)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
TUNABLE_INT_FETCH("debug.malloc.numzones",
|
|
|
|
|
&numzones);
|
|
|
|
|
|
|
|
|
|
/* Sanity check the number of malloc uma zones. */
|
|
|
|
|
if (numzones <= 0)
|
|
|
|
|
numzones = 1;
|
|
|
|
|
if (numzones > MALLOC_DEBUG_MAXZONES)
|
|
|
|
|
numzones = MALLOC_DEBUG_MAXZONES;
|
|
|
|
|
}
|
|
|
|
|
SYSINIT(numzones, SI_SUB_TUNABLES, SI_ORDER_ANY, tunable_set_numzones, NULL);
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_debug_malloc, OID_AUTO, numzones, CTLFLAG_RDTUN | CTLFLAG_NOFETCH,
|
2010-07-28 15:36:12 +00:00
|
|
|
&numzones, 0, "Number of malloc uma subzones");
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Any number that changes regularly is an okay choice for the
|
|
|
|
|
* offset. Build numbers are pretty good of you have them.
|
|
|
|
|
*/
|
|
|
|
|
static u_int zone_offset = __FreeBSD_version;
|
|
|
|
|
TUNABLE_INT("debug.malloc.zone_offset", &zone_offset);
|
|
|
|
|
SYSCTL_UINT(_debug_malloc, OID_AUTO, zone_offset, CTLFLAG_RDTUN,
|
|
|
|
|
&zone_offset, 0, "Separate malloc types by examining the "
|
|
|
|
|
"Nth character in the malloc type short description.");
|
|
|
|
|
|
2018-04-23 22:28:49 +00:00
|
|
|
static void
|
|
|
|
|
mtp_set_subzone(struct malloc_type *mtp)
|
2010-07-28 15:36:12 +00:00
|
|
|
{
|
2018-04-23 22:28:49 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
const char *desc;
|
2010-07-28 15:36:12 +00:00
|
|
|
size_t len;
|
|
|
|
|
u_int val;
|
|
|
|
|
|
2018-04-23 22:28:49 +00:00
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
desc = mtp->ks_shortdesc;
|
2010-07-28 15:36:12 +00:00
|
|
|
if (desc == NULL || (len = strlen(desc)) == 0)
|
2018-04-23 22:28:49 +00:00
|
|
|
val = 0;
|
|
|
|
|
else
|
|
|
|
|
val = desc[zone_offset % len];
|
|
|
|
|
mtip->mti_zone = (val % numzones);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline u_int
|
|
|
|
|
mtp_get_subzone(struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
|
|
|
|
|
KASSERT(mtip->mti_zone < numzones,
|
|
|
|
|
("mti_zone %u out of range %d",
|
|
|
|
|
mtip->mti_zone, numzones));
|
|
|
|
|
return (mtip->mti_zone);
|
2010-07-28 15:36:12 +00:00
|
|
|
}
|
|
|
|
|
#elif MALLOC_DEBUG_MAXZONES == 0
|
|
|
|
|
#error "MALLOC_DEBUG_MAXZONES must be positive."
|
|
|
|
|
#else
|
2018-04-23 22:28:49 +00:00
|
|
|
static void
|
|
|
|
|
mtp_set_subzone(struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
mtip->mti_zone = 0;
|
|
|
|
|
}
|
|
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
static inline u_int
|
2018-04-23 22:28:49 +00:00
|
|
|
mtp_get_subzone(struct malloc_type *mtp)
|
2010-07-28 15:36:12 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
#endif /* MALLOC_DEBUG_MAXZONES > 1 */
|
|
|
|
|
|
2002-11-01 18:58:12 +00:00
|
|
|
int
|
|
|
|
|
malloc_last_fail(void)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (time_uptime - t_malloc_fail);
|
|
|
|
|
}
|
|
|
|
|
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
/*
|
2006-07-23 19:51:39 +00:00
|
|
|
* An allocation has succeeded -- update malloc type statistics for the
|
|
|
|
|
* amount of bucket size. Occurs within a critical section so that the
|
|
|
|
|
* thread isn't preempted and doesn't migrate while updating per-PCU
|
|
|
|
|
* statistics.
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
*/
|
|
|
|
|
static void
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_zone_allocated(struct malloc_type *mtp, unsigned long size,
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
int zindx)
|
|
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type_stats *mtsp;
|
|
|
|
|
|
|
|
|
|
critical_enter();
|
|
|
|
|
mtip = mtp->ks_handle;
|
2018-09-23 19:00:06 +00:00
|
|
|
mtsp = zpcpu_get(mtip->mti_stats);
|
2005-07-27 23:17:31 +00:00
|
|
|
if (size > 0) {
|
|
|
|
|
mtsp->mts_memalloced += size;
|
|
|
|
|
mtsp->mts_numallocs++;
|
|
|
|
|
}
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
if (zindx != -1)
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtsp->mts_size |= 1 << zindx;
|
2008-05-23 00:43:36 +00:00
|
|
|
|
|
|
|
|
#ifdef KDTRACE_HOOKS
|
2018-04-24 01:06:20 +00:00
|
|
|
if (__predict_false(dtrace_malloc_enabled)) {
|
2008-05-23 00:43:36 +00:00
|
|
|
uint32_t probe_id = mtip->mti_probes[DTMALLOC_PROBE_MALLOC];
|
|
|
|
|
if (probe_id != 0)
|
|
|
|
|
(dtrace_malloc_probe)(probe_id,
|
|
|
|
|
(uintptr_t) mtp, (uintptr_t) mtip,
|
|
|
|
|
(uintptr_t) mtsp, size, zindx);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
critical_exit();
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_allocated(struct malloc_type *mtp, unsigned long size)
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
|
2005-07-27 23:17:31 +00:00
|
|
|
if (size > 0)
|
|
|
|
|
malloc_type_zone_allocated(mtp, size, -1);
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
2007-06-14 03:02:33 +00:00
|
|
|
* A free operation has occurred -- update malloc type statistics for the
|
2006-07-23 19:51:39 +00:00
|
|
|
* amount of the bucket size. Occurs within a critical section so that the
|
|
|
|
|
* thread isn't preempted and doesn't migrate while updating per-CPU
|
|
|
|
|
* statistics.
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
*/
|
|
|
|
|
void
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_freed(struct malloc_type *mtp, unsigned long size)
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type_stats *mtsp;
|
|
|
|
|
|
|
|
|
|
critical_enter();
|
|
|
|
|
mtip = mtp->ks_handle;
|
2018-09-23 19:00:06 +00:00
|
|
|
mtsp = zpcpu_get(mtip->mti_stats);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtsp->mts_memfreed += size;
|
|
|
|
|
mtsp->mts_numfrees++;
|
2008-05-23 00:43:36 +00:00
|
|
|
|
|
|
|
|
#ifdef KDTRACE_HOOKS
|
2018-04-24 01:06:20 +00:00
|
|
|
if (__predict_false(dtrace_malloc_enabled)) {
|
2008-05-23 00:43:36 +00:00
|
|
|
uint32_t probe_id = mtip->mti_probes[DTMALLOC_PROBE_FREE];
|
|
|
|
|
if (probe_id != 0)
|
|
|
|
|
(dtrace_malloc_probe)(probe_id,
|
|
|
|
|
(uintptr_t) mtp, (uintptr_t) mtip,
|
|
|
|
|
(uintptr_t) mtsp, size, 0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
critical_exit();
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
}
|
|
|
|
|
|
2011-10-27 02:52:24 +00:00
|
|
|
/*
|
|
|
|
|
* contigmalloc:
|
|
|
|
|
*
|
|
|
|
|
* Allocate a block of physically contiguous memory.
|
|
|
|
|
*
|
|
|
|
|
* If M_NOWAIT is set, this routine will not block and return NULL if
|
|
|
|
|
* the allocation fails.
|
|
|
|
|
*/
|
|
|
|
|
void *
|
|
|
|
|
contigmalloc(unsigned long size, struct malloc_type *type, int flags,
|
|
|
|
|
vm_paddr_t low, vm_paddr_t high, unsigned long alignment,
|
2012-03-01 19:58:34 +00:00
|
|
|
vm_paddr_t boundary)
|
2011-10-27 02:52:24 +00:00
|
|
|
{
|
|
|
|
|
void *ret;
|
|
|
|
|
|
2018-08-20 15:57:27 +00:00
|
|
|
ret = (void *)kmem_alloc_contig(size, flags, low, high, alignment,
|
|
|
|
|
boundary, VM_MEMATTR_DEFAULT);
|
2011-10-27 02:52:24 +00:00
|
|
|
if (ret != NULL)
|
|
|
|
|
malloc_type_allocated(type, round_page(size));
|
|
|
|
|
return (ret);
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
void *
|
2018-10-30 18:26:34 +00:00
|
|
|
contigmalloc_domainset(unsigned long size, struct malloc_type *type,
|
|
|
|
|
struct domainset *ds, int flags, vm_paddr_t low, vm_paddr_t high,
|
2018-01-12 23:25:05 +00:00
|
|
|
unsigned long alignment, vm_paddr_t boundary)
|
|
|
|
|
{
|
|
|
|
|
void *ret;
|
|
|
|
|
|
2018-10-30 18:26:34 +00:00
|
|
|
ret = (void *)kmem_alloc_contig_domainset(ds, size, flags, low, high,
|
2018-01-12 23:25:05 +00:00
|
|
|
alignment, boundary, VM_MEMATTR_DEFAULT);
|
|
|
|
|
if (ret != NULL)
|
|
|
|
|
malloc_type_allocated(type, round_page(size));
|
|
|
|
|
return (ret);
|
|
|
|
|
}
|
|
|
|
|
|
2011-10-27 02:52:24 +00:00
|
|
|
/*
|
|
|
|
|
* contigfree:
|
|
|
|
|
*
|
|
|
|
|
* Free a block of memory allocated by contigmalloc.
|
|
|
|
|
*
|
|
|
|
|
* This routine may not block.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
contigfree(void *addr, unsigned long size, struct malloc_type *type)
|
|
|
|
|
{
|
|
|
|
|
|
2018-08-25 19:38:08 +00:00
|
|
|
kmem_free((vm_offset_t)addr, size);
|
2011-10-27 02:52:24 +00:00
|
|
|
malloc_type_freed(type, round_page(size));
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
static int
|
2018-01-25 06:37:14 +00:00
|
|
|
malloc_dbg(caddr_t *vap, size_t *sizep, struct malloc_type *mtp,
|
2018-01-12 23:25:05 +00:00
|
|
|
int flags)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef INVARIANTS
|
2002-04-15 04:05:53 +00:00
|
|
|
int indx;
|
1997-10-10 14:06:34 +00:00
|
|
|
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC, ("malloc: bad malloc type magic"));
|
2003-03-10 19:39:53 +00:00
|
|
|
/*
|
2005-08-02 20:03:23 +00:00
|
|
|
* Check that exactly one of M_WAITOK or M_NOWAIT is specified.
|
2003-03-10 19:39:53 +00:00
|
|
|
*/
|
2005-08-02 20:03:23 +00:00
|
|
|
indx = flags & (M_WAITOK | M_NOWAIT);
|
2003-03-10 19:39:53 +00:00
|
|
|
if (indx != M_NOWAIT && indx != M_WAITOK) {
|
|
|
|
|
static struct timeval lasterr;
|
|
|
|
|
static int curerr, once;
|
|
|
|
|
if (once == 0 && ppsratecheck(&lasterr, &curerr, 1)) {
|
|
|
|
|
printf("Bad malloc flags: %x\n", indx);
|
2004-07-10 21:36:01 +00:00
|
|
|
kdb_backtrace();
|
2003-03-10 19:39:53 +00:00
|
|
|
flags |= M_WAITOK;
|
|
|
|
|
once++;
|
|
|
|
|
}
|
|
|
|
|
}
|
2003-03-10 20:24:54 +00:00
|
|
|
#endif
|
2003-03-26 20:18:40 +00:00
|
|
|
#ifdef MALLOC_MAKE_FAILURES
|
|
|
|
|
if ((flags & M_NOWAIT) && (malloc_failure_rate != 0)) {
|
|
|
|
|
atomic_add_int(&malloc_nowait_count, 1);
|
|
|
|
|
if ((malloc_nowait_count % malloc_failure_rate) == 0) {
|
|
|
|
|
atomic_add_int(&malloc_failure_count, 1);
|
2003-04-25 21:49:24 +00:00
|
|
|
t_malloc_fail = time_uptime;
|
2018-01-12 23:25:05 +00:00
|
|
|
*vap = NULL;
|
|
|
|
|
return (EJUSTRETURN);
|
2003-03-26 20:18:40 +00:00
|
|
|
}
|
|
|
|
|
}
|
2002-04-23 18:50:25 +00:00
|
|
|
#endif
|
2018-05-10 17:55:24 +00:00
|
|
|
if (flags & M_WAITOK) {
|
2001-09-12 08:38:13 +00:00
|
|
|
KASSERT(curthread->td_intr_nesting_level == 0,
|
2003-02-19 05:47:46 +00:00
|
|
|
("malloc(M_WAITOK) in interrupt context"));
|
2019-10-29 17:28:25 +00:00
|
|
|
if (__predict_false(!THREAD_CAN_SLEEP())) {
|
2019-10-15 21:24:25 +00:00
|
|
|
#ifdef EPOCH_TRACE
|
|
|
|
|
epoch_trace_list(curthread);
|
|
|
|
|
#endif
|
2019-10-29 17:28:25 +00:00
|
|
|
KASSERT(1,
|
|
|
|
|
("malloc(M_WAITOK) with sleeping prohibited"));
|
|
|
|
|
}
|
2018-05-10 17:55:24 +00:00
|
|
|
}
|
2015-12-11 20:05:07 +00:00
|
|
|
KASSERT(curthread->td_critnest == 0 || SCHEDULER_STOPPED(),
|
2015-11-19 14:04:53 +00:00
|
|
|
("malloc: called with spinlock or critical section held"));
|
|
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2018-01-12 23:25:05 +00:00
|
|
|
if (memguard_cmp_mtp(mtp, *sizep)) {
|
|
|
|
|
*vap = memguard_alloc(*sizep, flags);
|
|
|
|
|
if (*vap != NULL)
|
|
|
|
|
return (EJUSTRETURN);
|
2010-08-11 22:10:37 +00:00
|
|
|
/* This is unfortunate but should not be fatal. */
|
|
|
|
|
}
|
2005-01-21 18:09:17 +00:00
|
|
|
#endif
|
|
|
|
|
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
2018-01-12 23:25:05 +00:00
|
|
|
*sizep = redzone_size_ntor(*sizep);
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2019-11-29 03:14:10 +00:00
|
|
|
/*
|
|
|
|
|
* Handle large allocations and frees by using kmem_malloc directly.
|
|
|
|
|
*/
|
|
|
|
|
static inline bool
|
|
|
|
|
malloc_large_slab(uma_slab_t slab)
|
|
|
|
|
{
|
|
|
|
|
uintptr_t va;
|
|
|
|
|
|
|
|
|
|
va = (uintptr_t)slab;
|
|
|
|
|
return ((va & 1) != 0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline size_t
|
|
|
|
|
malloc_large_size(uma_slab_t slab)
|
|
|
|
|
{
|
|
|
|
|
uintptr_t va;
|
|
|
|
|
|
|
|
|
|
va = (uintptr_t)slab;
|
|
|
|
|
return (va >> 1);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static caddr_t
|
|
|
|
|
malloc_large(size_t *size, struct domainset *policy, int flags)
|
|
|
|
|
{
|
|
|
|
|
vm_offset_t va;
|
|
|
|
|
size_t sz;
|
|
|
|
|
|
|
|
|
|
sz = roundup(*size, PAGE_SIZE);
|
|
|
|
|
va = kmem_malloc_domainset(policy, sz, flags);
|
|
|
|
|
if (va != 0) {
|
|
|
|
|
/* The low bit is unused for slab pointers. */
|
|
|
|
|
vsetzoneslab(va, NULL, (void *)((sz << 1) | 1));
|
|
|
|
|
uma_total_inc(sz);
|
|
|
|
|
*size = sz;
|
|
|
|
|
}
|
|
|
|
|
return ((caddr_t)va);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
free_large(void *addr, size_t size)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
kmem_free((vm_offset_t)addr, size);
|
|
|
|
|
uma_total_dec(size);
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
/*
|
|
|
|
|
* malloc:
|
|
|
|
|
*
|
|
|
|
|
* Allocate a block of memory.
|
|
|
|
|
*
|
|
|
|
|
* If M_NOWAIT is set, this routine will not block and return NULL if
|
|
|
|
|
* the allocation fails.
|
|
|
|
|
*/
|
|
|
|
|
void *
|
2018-06-02 22:20:09 +00:00
|
|
|
(malloc)(size_t size, struct malloc_type *mtp, int flags)
|
2018-01-12 23:25:05 +00:00
|
|
|
{
|
|
|
|
|
int indx;
|
|
|
|
|
caddr_t va;
|
|
|
|
|
uma_zone_t zone;
|
|
|
|
|
#if defined(DEBUG_REDZONE)
|
|
|
|
|
unsigned long osize = size;
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
2018-05-19 04:43:49 +00:00
|
|
|
va = NULL;
|
2018-01-12 23:25:05 +00:00
|
|
|
if (malloc_dbg(&va, &size, mtp, flags) != 0)
|
|
|
|
|
return (va);
|
2006-01-31 11:09:21 +00:00
|
|
|
#endif
|
|
|
|
|
|
Make UMA and malloc(9) return non-executable memory in most cases.
Most kernel memory that is allocated after boot does not need to be
executable. There are a few exceptions. For example, kernel modules
do need executable memory, but they don't use UMA or malloc(9). The
BPF JIT compiler also needs executable memory and did use malloc(9)
until r317072.
(Note that a side effect of r316767 was that the "small allocation"
path in UMA on amd64 already returned non-executable memory. This
meant that some calls to malloc(9) or the UMA zone(9) allocator could
return executable memory, while others could return non-executable
memory. This change makes the behavior consistent.)
This change makes malloc(9) return non-executable memory unless the new
M_EXEC flag is specified. After this change, the UMA zone(9) allocator
will always return non-executable memory, and a KASSERT will catch
attempts to use the M_EXEC flag to allocate executable memory using
uma_zalloc() or its variants.
Allocations that do need executable memory have various choices. They
may use the M_EXEC flag to malloc(9), or they may use a different VM
interfact to obtain executable pages.
Now that malloc(9) again allows executable allocations, this change also
reverts most of r317072.
PR: 228927
Reviewed by: alc, kib, markj, jhb (previous version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D15691
2018-06-13 17:04:41 +00:00
|
|
|
if (size <= kmem_zmax && (flags & M_EXEC) == 0) {
|
2002-04-15 04:05:53 +00:00
|
|
|
if (size & KMEM_ZMASK)
|
|
|
|
|
size = (size & ~KMEM_ZMASK) + KMEM_ZBASE;
|
|
|
|
|
indx = kmemsize[size >> KMEM_ZSHIFT];
|
2018-04-23 22:28:49 +00:00
|
|
|
zone = kmemzones[indx].kz_zone[mtp_get_subzone(mtp)];
|
2002-04-15 04:05:53 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
|
|
|
|
krequests[size >> KMEM_ZSHIFT]++;
|
|
|
|
|
#endif
|
2002-03-19 09:11:49 +00:00
|
|
|
va = uma_zalloc(zone, flags);
|
Reimplement contigmalloc(9) with an algorithm which stands a greatly-
improved chance of working despite pressure from running programs.
Instead of trying to throw a bunch of pages out to swap and hope for
the best, only a range that can potentially fulfill contigmalloc(9)'s
request will have its contents paged out (potentially, not forcibly)
at a time.
The new contigmalloc operation still operates in three passes, but it
could potentially be tuned to more or less. The first pass only looks
at pages in the cache and free pages, so they would be thrown out
without having to block. If this is not enough, the subsequent passes
page out any unwired memory. To combat memory pressure refragmenting
the section of memory being laundered, each page is removed from the
systems' free memory queue once it has been freed so that blocking
later doesn't cause the memory laundered so far to get reallocated.
The page-out operations are now blocking, as it would make little sense
to try to push out a page, then get its status immediately afterward
to remove it from the available free pages queue, if it's unlikely to
have been freed. Another change is that if KVA allocation fails, the
allocated memory segment will be freed and not leaked.
There is a sysctl/tunable, defaulting to on, which causes the old
contigmalloc() algorithm to be used. Nonetheless, I have been using
vm.old_contigmalloc=0 for over a month. It is safe to switch at
run-time to see the difference it makes.
A new interface has been used which does not require mapping the
allocated pages into KVA: vm_page.h functions vm_page_alloc_contig()
and vm_page_release_contig(). These are what vm.old_contigmalloc=0
uses internally, so the sysctl/tunable does not affect their operation.
When using the contigmalloc(9) and contigfree(9) interfaces, memory
is now tracked with malloc(9) stats. Several functions have been
exported from kern_malloc.c to allow other subsystems to use these
statistics, as well. This invalidates the BUGS section of the
contigmalloc(9) manpage.
2004-07-19 06:21:27 +00:00
|
|
|
if (va != NULL)
|
2009-01-25 09:11:24 +00:00
|
|
|
size = zone->uz_size;
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_zone_allocated(mtp, va == NULL ? 0 : size, indx);
|
2002-03-19 09:11:49 +00:00
|
|
|
} else {
|
2019-11-29 03:14:10 +00:00
|
|
|
va = malloc_large(&size, DOMAINSET_RR(), flags);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_allocated(mtp, va == NULL ? 0 : size);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2003-05-12 05:09:56 +00:00
|
|
|
if (flags & M_WAITOK)
|
2003-02-19 05:47:46 +00:00
|
|
|
KASSERT(va != NULL, ("malloc(M_WAITOK) returned NULL"));
|
2003-05-12 05:09:56 +00:00
|
|
|
else if (va == NULL)
|
2002-11-01 18:58:12 +00:00
|
|
|
t_malloc_fail = time_uptime;
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
if (va != NULL)
|
|
|
|
|
va = redzone_setup(va, osize);
|
|
|
|
|
#endif
|
|
|
|
|
return ((void *) va);
|
|
|
|
|
}
|
|
|
|
|
|
2018-10-30 18:26:34 +00:00
|
|
|
static void *
|
2020-01-09 15:02:48 +00:00
|
|
|
malloc_domain(size_t *sizep, int *indxp, struct malloc_type *mtp, int domain,
|
2019-11-29 03:14:10 +00:00
|
|
|
int flags)
|
2018-01-12 23:25:05 +00:00
|
|
|
{
|
|
|
|
|
uma_zone_t zone;
|
2020-01-09 15:02:48 +00:00
|
|
|
caddr_t va;
|
|
|
|
|
size_t size;
|
|
|
|
|
int indx;
|
2018-01-12 23:25:05 +00:00
|
|
|
|
2020-01-09 15:02:48 +00:00
|
|
|
size = *sizep;
|
2019-11-29 03:14:10 +00:00
|
|
|
KASSERT(size <= kmem_zmax && (flags & M_EXEC) == 0,
|
|
|
|
|
("malloc_domain: Called with bad flag / size combination."));
|
|
|
|
|
if (size & KMEM_ZMASK)
|
|
|
|
|
size = (size & ~KMEM_ZMASK) + KMEM_ZBASE;
|
|
|
|
|
indx = kmemsize[size >> KMEM_ZSHIFT];
|
|
|
|
|
zone = kmemzones[indx].kz_zone[mtp_get_subzone(mtp)];
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
2019-11-29 03:14:10 +00:00
|
|
|
krequests[size >> KMEM_ZSHIFT]++;
|
2018-01-12 23:25:05 +00:00
|
|
|
#endif
|
2019-11-29 03:14:10 +00:00
|
|
|
va = uma_zalloc_domain(zone, NULL, domain, flags);
|
2006-01-31 11:09:21 +00:00
|
|
|
if (va != NULL)
|
2020-01-09 15:02:48 +00:00
|
|
|
*sizep = zone->uz_size;
|
2019-11-29 03:14:10 +00:00
|
|
|
*indxp = indx;
|
2020-01-09 15:02:48 +00:00
|
|
|
return ((void *)va);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
2018-10-30 18:26:34 +00:00
|
|
|
void *
|
|
|
|
|
malloc_domainset(size_t size, struct malloc_type *mtp, struct domainset *ds,
|
|
|
|
|
int flags)
|
|
|
|
|
{
|
|
|
|
|
struct vm_domainset_iter di;
|
2019-11-29 03:14:10 +00:00
|
|
|
caddr_t ret;
|
2018-10-30 18:26:34 +00:00
|
|
|
int domain;
|
2019-11-29 03:14:10 +00:00
|
|
|
int indx;
|
2018-10-30 18:26:34 +00:00
|
|
|
|
2019-11-29 03:14:10 +00:00
|
|
|
#if defined(DEBUG_REDZONE)
|
|
|
|
|
unsigned long osize = size;
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
ret= NULL;
|
|
|
|
|
if (malloc_dbg(&ret, &size, mtp, flags) != 0)
|
|
|
|
|
return (ret);
|
|
|
|
|
#endif
|
|
|
|
|
if (size <= kmem_zmax && (flags & M_EXEC) == 0) {
|
|
|
|
|
vm_domainset_iter_policy_init(&di, ds, &domain, &flags);
|
|
|
|
|
do {
|
2020-01-09 15:02:48 +00:00
|
|
|
ret = malloc_domain(&size, &indx, mtp, domain, flags);
|
2019-11-29 03:14:10 +00:00
|
|
|
} while (ret == NULL &&
|
|
|
|
|
vm_domainset_iter_policy(&di, &domain) == 0);
|
|
|
|
|
malloc_type_zone_allocated(mtp, ret == NULL ? 0 : size, indx);
|
|
|
|
|
} else {
|
|
|
|
|
/* Policy is handled by kmem. */
|
|
|
|
|
ret = malloc_large(&size, ds, flags);
|
|
|
|
|
malloc_type_allocated(mtp, ret == NULL ? 0 : size);
|
|
|
|
|
}
|
2018-10-30 18:26:34 +00:00
|
|
|
|
2019-11-29 03:14:10 +00:00
|
|
|
if (flags & M_WAITOK)
|
|
|
|
|
KASSERT(ret != NULL, ("malloc(M_WAITOK) returned NULL"));
|
|
|
|
|
else if (ret == NULL)
|
|
|
|
|
t_malloc_fail = time_uptime;
|
|
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
if (ret != NULL)
|
|
|
|
|
ret = redzone_setup(ret, osize);
|
|
|
|
|
#endif
|
2018-10-30 18:26:34 +00:00
|
|
|
return (ret);
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-07 13:21:01 +00:00
|
|
|
void *
|
|
|
|
|
mallocarray(size_t nmemb, size_t size, struct malloc_type *type, int flags)
|
|
|
|
|
{
|
|
|
|
|
|
2018-01-10 21:49:45 +00:00
|
|
|
if (WOULD_OVERFLOW(nmemb, size))
|
|
|
|
|
panic("mallocarray: %zu * %zu overflowed", nmemb, size);
|
2018-01-07 13:21:01 +00:00
|
|
|
|
|
|
|
|
return (malloc(size * nmemb, type, flags));
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
#ifdef INVARIANTS
|
|
|
|
|
static void
|
|
|
|
|
free_save_type(void *addr, struct malloc_type *mtp, u_long size)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2018-01-12 23:25:05 +00:00
|
|
|
struct malloc_type **mtpp = addr;
|
1997-10-10 14:06:34 +00:00
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
/*
|
|
|
|
|
* Cache a pointer to the malloc_type that most recently freed
|
|
|
|
|
* this memory here. This way we know who is most likely to
|
|
|
|
|
* have stepped on it later.
|
|
|
|
|
*
|
|
|
|
|
* This code assumes that size is a multiple of 8 bytes for
|
|
|
|
|
* 64 bit machines
|
|
|
|
|
*/
|
|
|
|
|
mtpp = (struct malloc_type **) ((unsigned long)mtpp & ~UMA_ALIGN_PTR);
|
|
|
|
|
mtpp += (size - sizeof(struct malloc_type *)) /
|
|
|
|
|
sizeof(struct malloc_type *);
|
|
|
|
|
*mtpp = mtp;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
static int
|
|
|
|
|
free_dbg(void **addrp, struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
void *addr;
|
|
|
|
|
|
|
|
|
|
addr = *addrp;
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC, ("free: bad malloc type magic"));
|
2015-12-11 20:05:07 +00:00
|
|
|
KASSERT(curthread->td_critnest == 0 || SCHEDULER_STOPPED(),
|
2015-11-19 14:04:53 +00:00
|
|
|
("free: called with spinlock or critical section held"));
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/* free(NULL, ...) does nothing */
|
|
|
|
|
if (addr == NULL)
|
2018-01-12 23:25:05 +00:00
|
|
|
return (EJUSTRETURN);
|
2002-03-13 01:42:33 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2010-08-11 22:10:37 +00:00
|
|
|
if (is_memguard_addr(addr)) {
|
2005-01-21 18:09:17 +00:00
|
|
|
memguard_free(addr);
|
2018-01-12 23:25:05 +00:00
|
|
|
return (EJUSTRETURN);
|
2005-01-21 18:09:17 +00:00
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
redzone_check(addr);
|
2018-01-12 23:25:05 +00:00
|
|
|
*addrp = redzone_addr_ntor(addr);
|
2006-01-31 11:09:21 +00:00
|
|
|
#endif
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
/*
|
|
|
|
|
* free:
|
|
|
|
|
*
|
|
|
|
|
* Free a block of memory allocated by malloc.
|
|
|
|
|
*
|
|
|
|
|
* This routine may not block.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
free(void *addr, struct malloc_type *mtp)
|
|
|
|
|
{
|
2019-11-28 07:49:25 +00:00
|
|
|
uma_zone_t zone;
|
2018-01-12 23:25:05 +00:00
|
|
|
uma_slab_t slab;
|
|
|
|
|
u_long size;
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
if (free_dbg(&addr, mtp) != 0)
|
|
|
|
|
return;
|
|
|
|
|
#endif
|
|
|
|
|
/* free(NULL, ...) does nothing */
|
|
|
|
|
if (addr == NULL)
|
|
|
|
|
return;
|
|
|
|
|
|
2019-11-28 07:49:25 +00:00
|
|
|
vtozoneslab((vm_offset_t)addr & (~UMA_SLAB_MASK), &zone, &slab);
|
2002-03-19 09:11:49 +00:00
|
|
|
if (slab == NULL)
|
2002-04-15 04:05:53 +00:00
|
|
|
panic("free: address %p(%p) has not been allocated.\n",
|
2002-09-18 08:26:30 +00:00
|
|
|
addr, (void *)((u_long)addr & (~UMA_SLAB_MASK)));
|
|
|
|
|
|
2019-11-29 03:14:10 +00:00
|
|
|
if (__predict_true(!malloc_large_slab(slab))) {
|
2019-11-28 07:49:25 +00:00
|
|
|
size = zone->uz_size;
|
2002-05-02 09:07:04 +00:00
|
|
|
#ifdef INVARIANTS
|
2018-01-12 23:25:05 +00:00
|
|
|
free_save_type(addr, mtp, size);
|
|
|
|
|
#endif
|
2019-11-28 07:49:25 +00:00
|
|
|
uma_zfree_arg(zone, addr, slab);
|
2018-01-12 23:25:05 +00:00
|
|
|
} else {
|
2019-11-29 03:14:10 +00:00
|
|
|
size = malloc_large_size(slab);
|
|
|
|
|
free_large(addr, size);
|
2018-01-12 23:25:05 +00:00
|
|
|
}
|
|
|
|
|
malloc_type_freed(mtp, size);
|
|
|
|
|
}
|
|
|
|
|
|
2020-02-16 00:12:53 +00:00
|
|
|
/*
|
|
|
|
|
* zfree:
|
|
|
|
|
*
|
|
|
|
|
* Zero then free a block of memory allocated by malloc.
|
|
|
|
|
*
|
|
|
|
|
* This routine may not block.
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
zfree(void *addr, struct malloc_type *mtp)
|
|
|
|
|
{
|
|
|
|
|
uma_zone_t zone;
|
|
|
|
|
uma_slab_t slab;
|
|
|
|
|
u_long size;
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
if (free_dbg(&addr, mtp) != 0)
|
|
|
|
|
return;
|
|
|
|
|
#endif
|
|
|
|
|
/* free(NULL, ...) does nothing */
|
|
|
|
|
if (addr == NULL)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
vtozoneslab((vm_offset_t)addr & (~UMA_SLAB_MASK), &zone, &slab);
|
|
|
|
|
if (slab == NULL)
|
|
|
|
|
panic("free: address %p(%p) has not been allocated.\n",
|
|
|
|
|
addr, (void *)((u_long)addr & (~UMA_SLAB_MASK)));
|
|
|
|
|
|
|
|
|
|
if (__predict_true(!malloc_large_slab(slab))) {
|
|
|
|
|
size = zone->uz_size;
|
|
|
|
|
#ifdef INVARIANTS
|
|
|
|
|
free_save_type(addr, mtp, size);
|
|
|
|
|
#endif
|
|
|
|
|
explicit_bzero(addr, size);
|
|
|
|
|
uma_zfree_arg(zone, addr, slab);
|
|
|
|
|
} else {
|
|
|
|
|
size = malloc_large_size(slab);
|
|
|
|
|
explicit_bzero(addr, size);
|
|
|
|
|
free_large(addr, size);
|
|
|
|
|
}
|
|
|
|
|
malloc_type_freed(mtp, size);
|
|
|
|
|
}
|
|
|
|
|
|
2018-01-12 23:25:05 +00:00
|
|
|
void
|
|
|
|
|
free_domain(void *addr, struct malloc_type *mtp)
|
|
|
|
|
{
|
2019-11-28 07:49:25 +00:00
|
|
|
uma_zone_t zone;
|
2018-01-12 23:25:05 +00:00
|
|
|
uma_slab_t slab;
|
|
|
|
|
u_long size;
|
|
|
|
|
|
|
|
|
|
#ifdef MALLOC_DEBUG
|
|
|
|
|
if (free_dbg(&addr, mtp) != 0)
|
|
|
|
|
return;
|
2002-05-02 09:07:04 +00:00
|
|
|
#endif
|
2018-01-12 23:25:05 +00:00
|
|
|
|
|
|
|
|
/* free(NULL, ...) does nothing */
|
|
|
|
|
if (addr == NULL)
|
|
|
|
|
return;
|
|
|
|
|
|
2019-11-28 07:49:25 +00:00
|
|
|
vtozoneslab((vm_offset_t)addr & (~UMA_SLAB_MASK), &zone, &slab);
|
2018-01-12 23:25:05 +00:00
|
|
|
if (slab == NULL)
|
|
|
|
|
panic("free_domain: address %p(%p) has not been allocated.\n",
|
|
|
|
|
addr, (void *)((u_long)addr & (~UMA_SLAB_MASK)));
|
|
|
|
|
|
2019-11-29 03:14:10 +00:00
|
|
|
if (__predict_true(!malloc_large_slab(slab))) {
|
2019-11-28 07:49:25 +00:00
|
|
|
size = zone->uz_size;
|
2002-05-02 09:07:04 +00:00
|
|
|
#ifdef INVARIANTS
|
2018-01-12 23:25:05 +00:00
|
|
|
free_save_type(addr, mtp, size);
|
2002-05-02 09:07:04 +00:00
|
|
|
#endif
|
2019-11-28 07:49:25 +00:00
|
|
|
uma_zfree_domain(zone, addr, slab);
|
2002-03-19 09:11:49 +00:00
|
|
|
} else {
|
2019-11-29 03:14:10 +00:00
|
|
|
size = malloc_large_size(slab);
|
|
|
|
|
free_large(addr, size);
|
1999-05-06 18:13:11 +00:00
|
|
|
}
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
malloc_type_freed(mtp, size);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/*
|
|
|
|
|
* realloc: change the size of a memory block
|
|
|
|
|
*/
|
|
|
|
|
void *
|
2018-01-24 19:37:18 +00:00
|
|
|
realloc(void *addr, size_t size, struct malloc_type *mtp, int flags)
|
2002-03-13 01:42:33 +00:00
|
|
|
{
|
2019-11-28 07:49:25 +00:00
|
|
|
uma_zone_t zone;
|
2002-03-19 09:11:49 +00:00
|
|
|
uma_slab_t slab;
|
2002-03-13 01:42:33 +00:00
|
|
|
unsigned long alloc;
|
|
|
|
|
void *newaddr;
|
|
|
|
|
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC,
|
|
|
|
|
("realloc: bad malloc type magic"));
|
2015-12-11 20:05:07 +00:00
|
|
|
KASSERT(curthread->td_critnest == 0 || SCHEDULER_STOPPED(),
|
2015-11-19 14:04:53 +00:00
|
|
|
("realloc: called with spinlock or critical section held"));
|
|
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/* realloc(NULL, ...) is equivalent to malloc(...) */
|
|
|
|
|
if (addr == NULL)
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
return (malloc(size, mtp, flags));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* XXX: Should report free of old memory and alloc of new memory to
|
|
|
|
|
* per-CPU stats.
|
|
|
|
|
*/
|
2002-03-13 01:42:33 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2010-08-31 16:57:58 +00:00
|
|
|
if (is_memguard_addr(addr))
|
|
|
|
|
return (memguard_realloc(addr, size, mtp, flags));
|
2005-01-21 18:09:17 +00:00
|
|
|
#endif
|
|
|
|
|
|
2006-01-31 11:09:21 +00:00
|
|
|
#ifdef DEBUG_REDZONE
|
|
|
|
|
slab = NULL;
|
2019-11-28 08:56:14 +00:00
|
|
|
zone = NULL;
|
2006-01-31 11:09:21 +00:00
|
|
|
alloc = redzone_get_size(addr);
|
|
|
|
|
#else
|
2019-11-28 07:49:25 +00:00
|
|
|
vtozoneslab((vm_offset_t)addr & (~UMA_SLAB_MASK), &zone, &slab);
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2002-03-13 01:42:33 +00:00
|
|
|
/* Sanity check */
|
2002-03-19 09:11:49 +00:00
|
|
|
KASSERT(slab != NULL,
|
2002-03-13 01:42:33 +00:00
|
|
|
("realloc: address %p out of range", (void *)addr));
|
|
|
|
|
|
|
|
|
|
/* Get the size of the original block */
|
2019-11-29 03:14:10 +00:00
|
|
|
if (!malloc_large_slab(slab))
|
2019-11-28 07:49:25 +00:00
|
|
|
alloc = zone->uz_size;
|
2002-03-19 09:11:49 +00:00
|
|
|
else
|
2019-11-29 03:14:10 +00:00
|
|
|
alloc = malloc_large_size(slab);
|
2002-03-13 01:42:33 +00:00
|
|
|
|
|
|
|
|
/* Reuse the original block if appropriate */
|
|
|
|
|
if (size <= alloc
|
|
|
|
|
&& (size > (alloc >> REALLOC_FRACTION) || alloc == MINALLOCSIZE))
|
|
|
|
|
return (addr);
|
2006-01-31 11:09:21 +00:00
|
|
|
#endif /* !DEBUG_REDZONE */
|
2002-03-13 01:42:33 +00:00
|
|
|
|
|
|
|
|
/* Allocate a new, bigger (or smaller) block */
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
if ((newaddr = malloc(size, mtp, flags)) == NULL)
|
2002-03-13 01:42:33 +00:00
|
|
|
return (NULL);
|
|
|
|
|
|
|
|
|
|
/* Copy over original contents */
|
|
|
|
|
bcopy(addr, newaddr, min(size, alloc));
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
free(addr, mtp);
|
2002-03-13 01:42:33 +00:00
|
|
|
return (newaddr);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* reallocf: same as realloc() but free memory on failure.
|
|
|
|
|
*/
|
|
|
|
|
void *
|
2018-01-24 19:37:18 +00:00
|
|
|
reallocf(void *addr, size_t size, struct malloc_type *mtp, int flags)
|
2002-03-13 01:42:33 +00:00
|
|
|
{
|
|
|
|
|
void *mem;
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
if ((mem = realloc(addr, size, mtp, flags)) == NULL)
|
|
|
|
|
free(addr, mtp);
|
2002-03-13 01:42:33 +00:00
|
|
|
return (mem);
|
|
|
|
|
}
|
|
|
|
|
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
CTASSERT(VM_KMEM_SIZE_SCALE >= 1);
|
|
|
|
|
|
2013-08-07 06:21:20 +00:00
|
|
|
/*
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* Initialize the kernel memory (kmem) arena.
|
2013-08-07 06:21:20 +00:00
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
kmeminit(void)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2014-06-28 03:56:17 +00:00
|
|
|
u_long mem_size;
|
|
|
|
|
u_long tmp;
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
|
2014-06-28 03:56:17 +00:00
|
|
|
#ifdef VM_KMEM_SIZE
|
|
|
|
|
if (vm_kmem_size == 0)
|
|
|
|
|
vm_kmem_size = VM_KMEM_SIZE;
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef VM_KMEM_SIZE_MIN
|
|
|
|
|
if (vm_kmem_size_min == 0)
|
|
|
|
|
vm_kmem_size_min = VM_KMEM_SIZE_MIN;
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef VM_KMEM_SIZE_MAX
|
|
|
|
|
if (vm_kmem_size_max == 0)
|
|
|
|
|
vm_kmem_size_max = VM_KMEM_SIZE_MAX;
|
|
|
|
|
#endif
|
1998-02-23 07:41:23 +00:00
|
|
|
/*
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* Calculate the amount of kernel virtual address (KVA) space that is
|
|
|
|
|
* preallocated to the kmem arena. In order to support a wide range
|
|
|
|
|
* of machines, it is a function of the physical memory size,
|
|
|
|
|
* specifically,
|
|
|
|
|
*
|
|
|
|
|
* min(max(physical memory size / VM_KMEM_SIZE_SCALE,
|
|
|
|
|
* VM_KMEM_SIZE_MIN), VM_KMEM_SIZE_MAX)
|
1998-02-23 07:41:23 +00:00
|
|
|
*
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* Every architecture must define an integral value for
|
|
|
|
|
* VM_KMEM_SIZE_SCALE. However, the definitions of VM_KMEM_SIZE_MIN
|
|
|
|
|
* and VM_KMEM_SIZE_MAX, which represent respectively the floor and
|
|
|
|
|
* ceiling on this preallocation, are optional. Typically,
|
|
|
|
|
* VM_KMEM_SIZE_MAX is itself a function of the available KVA space on
|
|
|
|
|
* a given architecture.
|
1998-02-23 07:41:23 +00:00
|
|
|
*/
|
2014-03-22 10:26:09 +00:00
|
|
|
mem_size = vm_cnt.v_page_count;
|
2014-09-22 05:07:22 +00:00
|
|
|
if (mem_size <= 32768) /* delphij XXX 128MB */
|
|
|
|
|
kmem_zmax = PAGE_SIZE;
|
1998-02-23 07:41:23 +00:00
|
|
|
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
if (vm_kmem_size_scale < 1)
|
|
|
|
|
vm_kmem_size_scale = VM_KMEM_SIZE_SCALE;
|
|
|
|
|
|
2014-06-28 03:56:17 +00:00
|
|
|
/*
|
|
|
|
|
* Check if we should use defaults for the "vm_kmem_size"
|
|
|
|
|
* variable:
|
|
|
|
|
*/
|
|
|
|
|
if (vm_kmem_size == 0) {
|
2019-01-14 07:31:19 +00:00
|
|
|
vm_kmem_size = mem_size / vm_kmem_size_scale;
|
|
|
|
|
vm_kmem_size = vm_kmem_size * PAGE_SIZE < vm_kmem_size ?
|
|
|
|
|
vm_kmem_size_max : vm_kmem_size * PAGE_SIZE;
|
2014-06-28 03:56:17 +00:00
|
|
|
if (vm_kmem_size_min > 0 && vm_kmem_size < vm_kmem_size_min)
|
|
|
|
|
vm_kmem_size = vm_kmem_size_min;
|
|
|
|
|
if (vm_kmem_size_max > 0 && vm_kmem_size >= vm_kmem_size_max)
|
|
|
|
|
vm_kmem_size = vm_kmem_size_max;
|
|
|
|
|
}
|
2019-01-14 07:31:19 +00:00
|
|
|
if (vm_kmem_size == 0)
|
|
|
|
|
panic("Tune VM_KMEM_SIZE_* for the platform");
|
1998-02-23 07:41:23 +00:00
|
|
|
|
2000-01-28 04:04:58 +00:00
|
|
|
/*
|
2014-06-28 03:56:17 +00:00
|
|
|
* The amount of KVA space that is preallocated to the
|
As of r257209, all architectures have defined VM_KMEM_SIZE_SCALE. In other
words, every architecture is now auto-sizing the kmem arena. This revision
changes kmeminit() so that the definition of VM_KMEM_SIZE_SCALE becomes
mandatory and the definition of VM_KMEM_SIZE becomes optional.
Replace or eliminate all existing definitions of VM_KMEM_SIZE. With
auto-sizing enabled, VM_KMEM_SIZE effectively became an alternate spelling
for VM_KMEM_SIZE_MIN on most architectures. Use VM_KMEM_SIZE_MIN for
clarity.
Change kmeminit() so that the effect of defining VM_KMEM_SIZE is similar to
that of setting the tunable vm.kmem_size. Whereas the macros
VM_KMEM_SIZE_{MAX,MIN,SCALE} have had the same effect as the tunables
vm.kmem_size_{max,min,scale}, the effects of VM_KMEM_SIZE and vm.kmem_size
have been distinct. In particular, whereas VM_KMEM_SIZE was overridden by
VM_KMEM_SIZE_{MAX,MIN,SCALE} and vm.kmem_size_{max,min,scale}, vm.kmem_size
was not. Remedy this inconsistency. Now, VM_KMEM_SIZE can be used to set
the size of the kmem arena at compile-time without that value being
overridden by auto-sizing.
Update the nearby comments to reflect the kmem submap being replaced by the
kmem arena. Stop duplicating the auto-sizing formula in every machine-
dependent vmparam.h and place it in kmeminit() where auto-sizing takes
place.
Reviewed by: kib (an earlier version)
Sponsored by: EMC / Isilon Storage Division
2013-11-08 16:25:00 +00:00
|
|
|
* kmem arena can be set statically at compile-time or manually
|
|
|
|
|
* through the kernel environment. However, it is still limited to
|
|
|
|
|
* twice the physical memory size, which has been sufficient to handle
|
|
|
|
|
* the most severe cases of external fragmentation in the kmem arena.
|
2000-01-28 04:04:58 +00:00
|
|
|
*/
|
2011-12-07 07:03:14 +00:00
|
|
|
if (vm_kmem_size / 2 / PAGE_SIZE > mem_size)
|
|
|
|
|
vm_kmem_size = 2 * mem_size * PAGE_SIZE;
|
1998-02-23 07:41:23 +00:00
|
|
|
|
2013-08-09 22:30:54 +00:00
|
|
|
vm_kmem_size = round_page(vm_kmem_size);
|
2010-08-11 22:10:37 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
2012-07-15 20:29:48 +00:00
|
|
|
tmp = memguard_fudge(vm_kmem_size, kernel_map);
|
2010-08-11 22:10:37 +00:00
|
|
|
#else
|
|
|
|
|
tmp = vm_kmem_size;
|
|
|
|
|
#endif
|
2017-11-28 23:40:54 +00:00
|
|
|
uma_set_limit(tmp);
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2005-01-21 18:09:17 +00:00
|
|
|
#ifdef DEBUG_MEMGUARD
|
|
|
|
|
/*
|
|
|
|
|
* Initialize MemGuard if support compiled in. MemGuard is a
|
|
|
|
|
* replacement allocator used for detecting tamper-after-free
|
|
|
|
|
* scenarios as they occur. It is only used for debugging.
|
|
|
|
|
*/
|
2017-11-28 23:40:54 +00:00
|
|
|
memguard_init(kernel_arena);
|
2005-01-21 18:09:17 +00:00
|
|
|
#endif
|
2013-08-07 06:21:20 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initialize the kernel memory allocator
|
|
|
|
|
*/
|
|
|
|
|
/* ARGSUSED*/
|
|
|
|
|
static void
|
|
|
|
|
mallocinit(void *dummy)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
uint8_t indx;
|
|
|
|
|
|
|
|
|
|
mtx_init(&malloc_mtx, "malloc", NULL, MTX_DEF);
|
|
|
|
|
|
|
|
|
|
kmeminit();
|
2005-01-21 18:09:17 +00:00
|
|
|
|
2014-08-14 05:31:39 +00:00
|
|
|
if (kmem_zmax < PAGE_SIZE || kmem_zmax > KMEM_ZMAX)
|
|
|
|
|
kmem_zmax = KMEM_ZMAX;
|
|
|
|
|
|
2018-09-23 19:00:06 +00:00
|
|
|
mt_stats_zone = uma_zcreate("mt_stats_zone",
|
|
|
|
|
sizeof(struct malloc_type_stats), NULL, NULL, NULL, NULL,
|
|
|
|
|
UMA_ALIGN_PTR, UMA_ZONE_PCPU);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mt_zone = uma_zcreate("mt_zone", sizeof(struct malloc_type_internal),
|
|
|
|
|
#ifdef INVARIANTS
|
|
|
|
|
mtrash_ctor, mtrash_dtor, mtrash_init, mtrash_fini,
|
|
|
|
|
#else
|
|
|
|
|
NULL, NULL, NULL, NULL,
|
|
|
|
|
#endif
|
|
|
|
|
UMA_ALIGN_PTR, UMA_ZONE_MALLOC);
|
2002-04-15 04:05:53 +00:00
|
|
|
for (i = 0, indx = 0; kmemzones[indx].kz_size != 0; indx++) {
|
|
|
|
|
int size = kmemzones[indx].kz_size;
|
2020-02-23 03:32:04 +00:00
|
|
|
const char *name = kmemzones[indx].kz_name;
|
2010-07-28 15:36:12 +00:00
|
|
|
int subzone;
|
2002-03-19 09:11:49 +00:00
|
|
|
|
2010-07-28 15:36:12 +00:00
|
|
|
for (subzone = 0; subzone < numzones; subzone++) {
|
|
|
|
|
kmemzones[indx].kz_zone[subzone] =
|
|
|
|
|
uma_zcreate(name, size,
|
2002-04-30 07:54:25 +00:00
|
|
|
#ifdef INVARIANTS
|
2010-07-28 15:36:12 +00:00
|
|
|
mtrash_ctor, mtrash_dtor, mtrash_init, mtrash_fini,
|
2002-04-30 07:54:25 +00:00
|
|
|
#else
|
2010-07-28 15:36:12 +00:00
|
|
|
NULL, NULL, NULL, NULL,
|
2002-04-30 07:54:25 +00:00
|
|
|
#endif
|
2010-07-28 15:36:12 +00:00
|
|
|
UMA_ALIGN_PTR, UMA_ZONE_MALLOC);
|
|
|
|
|
}
|
2002-03-19 09:11:49 +00:00
|
|
|
for (;i <= size; i+= KMEM_ZBASE)
|
2002-04-15 04:05:53 +00:00
|
|
|
kmemsize[i >> KMEM_ZSHIFT] = indx;
|
2014-08-14 05:31:39 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1997-10-10 14:06:34 +00:00
|
|
|
}
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSINIT(kmem, SI_SUB_KMEM, SI_ORDER_SECOND, mallocinit, NULL);
|
1997-10-10 14:06:34 +00:00
|
|
|
|
1998-11-10 08:46:24 +00:00
|
|
|
void
|
2005-04-12 23:54:34 +00:00
|
|
|
malloc_init(void *data)
|
1997-10-10 14:06:34 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type *mtp;
|
1997-10-10 14:06:34 +00:00
|
|
|
|
2014-03-22 10:26:09 +00:00
|
|
|
KASSERT(vm_cnt.v_page_count != 0, ("malloc_register before vm_init"));
|
1997-10-28 19:01:02 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtp = data;
|
2009-06-05 09:16:52 +00:00
|
|
|
if (mtp->ks_magic != M_MAGIC)
|
|
|
|
|
panic("malloc_init: bad malloc type magic");
|
2009-04-19 12:41:37 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtip = uma_zalloc(mt_zone, M_WAITOK | M_ZERO);
|
2018-09-23 19:00:06 +00:00
|
|
|
mtip->mti_stats = uma_zalloc_pcpu(mt_stats_zone, M_WAITOK | M_ZERO);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtp->ks_handle = mtip;
|
2018-04-23 22:28:49 +00:00
|
|
|
mtp_set_subzone(mtp);
|
1997-12-05 05:36:58 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
mtp->ks_next = kmemstatistics;
|
|
|
|
|
kmemstatistics = mtp;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
kmemcount++;
|
2002-04-15 04:05:53 +00:00
|
|
|
mtx_unlock(&malloc_mtx);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1998-11-10 08:46:24 +00:00
|
|
|
|
|
|
|
|
void
|
2005-04-12 23:54:34 +00:00
|
|
|
malloc_uninit(void *data)
|
1998-11-10 08:46:24 +00:00
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type_internal *mtip;
|
2005-11-03 13:48:59 +00:00
|
|
|
struct malloc_type_stats *mtsp;
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct malloc_type *mtp, *temp;
|
2006-03-03 22:36:52 +00:00
|
|
|
uma_slab_t slab;
|
2005-11-03 13:48:59 +00:00
|
|
|
long temp_allocs, temp_bytes;
|
|
|
|
|
int i;
|
1998-11-10 08:46:24 +00:00
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtp = data;
|
2009-04-19 12:41:37 +00:00
|
|
|
KASSERT(mtp->ks_magic == M_MAGIC,
|
|
|
|
|
("malloc_uninit: bad malloc type magic"));
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
KASSERT(mtp->ks_handle != NULL, ("malloc_deregister: cookie NULL"));
|
2009-04-19 12:41:37 +00:00
|
|
|
|
2002-04-15 04:05:53 +00:00
|
|
|
mtx_lock(&malloc_mtx);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
mtp->ks_handle = NULL;
|
|
|
|
|
if (mtp != kmemstatistics) {
|
|
|
|
|
for (temp = kmemstatistics; temp != NULL;
|
|
|
|
|
temp = temp->ks_next) {
|
2009-06-05 09:16:52 +00:00
|
|
|
if (temp->ks_next == mtp) {
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
temp->ks_next = mtp->ks_next;
|
2009-06-05 09:16:52 +00:00
|
|
|
break;
|
|
|
|
|
}
|
1998-11-10 08:46:24 +00:00
|
|
|
}
|
2009-06-05 09:16:52 +00:00
|
|
|
KASSERT(temp,
|
|
|
|
|
("malloc_uninit: type '%s' not found", mtp->ks_shortdesc));
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
} else
|
|
|
|
|
kmemstatistics = mtp->ks_next;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
kmemcount--;
|
2002-04-15 04:05:53 +00:00
|
|
|
mtx_unlock(&malloc_mtx);
|
2005-11-03 13:48:59 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Look for memory leaks.
|
|
|
|
|
*/
|
|
|
|
|
temp_allocs = temp_bytes = 0;
|
2018-09-23 19:00:06 +00:00
|
|
|
for (i = 0; i <= mp_maxid; i++) {
|
|
|
|
|
mtsp = zpcpu_get_cpu(mtip->mti_stats, i);
|
2005-11-03 13:48:59 +00:00
|
|
|
temp_allocs += mtsp->mts_numallocs;
|
|
|
|
|
temp_allocs -= mtsp->mts_numfrees;
|
|
|
|
|
temp_bytes += mtsp->mts_memalloced;
|
|
|
|
|
temp_bytes -= mtsp->mts_memfreed;
|
|
|
|
|
}
|
|
|
|
|
if (temp_allocs > 0 || temp_bytes > 0) {
|
|
|
|
|
printf("Warning: memory type %s leaked memory on destroy "
|
|
|
|
|
"(%ld allocations, %ld bytes leaked).\n", mtp->ks_shortdesc,
|
|
|
|
|
temp_allocs, temp_bytes);
|
|
|
|
|
}
|
|
|
|
|
|
2006-03-03 22:36:52 +00:00
|
|
|
slab = vtoslab((vm_offset_t) mtip & (~UMA_SLAB_MASK));
|
2018-09-23 19:00:06 +00:00
|
|
|
uma_zfree_pcpu(mt_stats_zone, mtip->mti_stats);
|
2006-03-03 22:36:52 +00:00
|
|
|
uma_zfree_arg(mt_zone, mtip, slab);
|
1998-11-10 08:46:24 +00:00
|
|
|
}
|
2002-04-15 04:05:53 +00:00
|
|
|
|
2005-12-30 11:45:07 +00:00
|
|
|
struct malloc_type *
|
|
|
|
|
malloc_desc2type(const char *desc)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type *mtp;
|
|
|
|
|
|
|
|
|
|
mtx_assert(&malloc_mtx, MA_OWNED);
|
|
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
if (strcmp(mtp->ks_shortdesc, desc) == 0)
|
|
|
|
|
return (mtp);
|
|
|
|
|
}
|
|
|
|
|
return (NULL);
|
|
|
|
|
}
|
|
|
|
|
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
static int
|
|
|
|
|
sysctl_kern_malloc_stats(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_stream_header mtsh;
|
|
|
|
|
struct malloc_type_internal *mtip;
|
2018-09-23 19:00:06 +00:00
|
|
|
struct malloc_type_stats *mtsp, zeromts;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
struct malloc_type_header mth;
|
|
|
|
|
struct malloc_type *mtp;
|
2010-09-16 16:13:12 +00:00
|
|
|
int error, i;
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
struct sbuf sbuf;
|
|
|
|
|
|
2011-01-27 00:34:12 +00:00
|
|
|
error = sysctl_wire_old_buffer(req, 0);
|
|
|
|
|
if (error != 0)
|
|
|
|
|
return (error);
|
2010-09-16 16:13:12 +00:00
|
|
|
sbuf_new_for_sysctl(&sbuf, NULL, 128, req);
|
2015-03-14 17:08:28 +00:00
|
|
|
sbuf_clear_flags(&sbuf, SBUF_INCLUDENUL);
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
|
2018-09-23 19:00:06 +00:00
|
|
|
bzero(&zeromts, sizeof(zeromts));
|
|
|
|
|
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
/*
|
|
|
|
|
* Insert stream header.
|
|
|
|
|
*/
|
|
|
|
|
bzero(&mtsh, sizeof(mtsh));
|
|
|
|
|
mtsh.mtsh_version = MALLOC_TYPE_STREAM_VERSION;
|
|
|
|
|
mtsh.mtsh_maxcpus = MAXCPU;
|
|
|
|
|
mtsh.mtsh_count = kmemcount;
|
2010-09-16 16:13:12 +00:00
|
|
|
(void)sbuf_bcat(&sbuf, &mtsh, sizeof(mtsh));
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert alternating sequence of type headers and type statistics.
|
|
|
|
|
*/
|
|
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
mtip = (struct malloc_type_internal *)mtp->ks_handle;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert type header.
|
|
|
|
|
*/
|
|
|
|
|
bzero(&mth, sizeof(mth));
|
|
|
|
|
strlcpy(mth.mth_name, mtp->ks_shortdesc, MALLOC_MAX_NAME);
|
2010-09-16 16:13:12 +00:00
|
|
|
(void)sbuf_bcat(&sbuf, &mth, sizeof(mth));
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Insert type statistics for each CPU.
|
|
|
|
|
*/
|
2018-09-23 19:00:06 +00:00
|
|
|
for (i = 0; i <= mp_maxid; i++) {
|
|
|
|
|
mtsp = zpcpu_get_cpu(mtip->mti_stats, i);
|
|
|
|
|
(void)sbuf_bcat(&sbuf, mtsp, sizeof(*mtsp));
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
}
|
2018-09-23 19:00:06 +00:00
|
|
|
/*
|
|
|
|
|
* Fill in the missing CPUs.
|
|
|
|
|
*/
|
|
|
|
|
for (; i < MAXCPU; i++) {
|
|
|
|
|
(void)sbuf_bcat(&sbuf, &zeromts, sizeof(zeromts));
|
|
|
|
|
}
|
|
|
|
|
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
}
|
|
|
|
|
mtx_unlock(&malloc_mtx);
|
2010-09-16 16:13:12 +00:00
|
|
|
error = sbuf_finish(&sbuf);
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
sbuf_delete(&sbuf);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2020-02-26 14:26:36 +00:00
|
|
|
SYSCTL_PROC(_kern, OID_AUTO, malloc_stats,
|
|
|
|
|
CTLFLAG_RD | CTLTYPE_STRUCT | CTLFLAG_MPSAFE, 0, 0,
|
|
|
|
|
sysctl_kern_malloc_stats, "s,malloc_type_ustats",
|
Introduce a new sysctl, kern.malloc_stats, which exports kernel malloc
statistics via a binary structure stream:
- Add structure 'malloc_type_stream_header', which defines a stream
version, definition of MAXCPUS used in the stream, and a number of
malloc_type records in the stream.
- Add structure 'malloc_type_header', which defines the name of the
malloc type being reported on.
- When the sysctl is queried, return a stream header, followed by a
series of type descriptions, each consisting of a type header
followed by a series of MAXCPUS malloc_type_stats structures holding
per-CPU allocation information. Typical values of MAXCPUS will be 1
(UP compiled kernel) and 16 (SMP compiled kernel).
This query mechanism allows user space monitoring tools to extract
memory allocation statistics in a machine-readable form, and to do so
at a per-CPU granularity, allowing monitoring of allocation patterns
across CPUs in order to better understand the distribution of work and
memory flow over multiple CPUs.
While here:
- Bump statistics width to uint64_t, and hard code using fixed-width
type in order to be more sure about structure layout in the stream.
We allocate and free a lot of memory.
- Add kmemcount, a counter of the number of registered malloc types,
in order to avoid excessive manual counting of types. Export via a
new sysctl to allow user-space code to better size buffers.
- De-XXX comment on no longer maintaining the high watermark in old
sysctl monitoring code.
A follow-up commit of libmemstat(3), a library to monitor kernel memory
allocation, will occur in the next few days. Likewise, similar changes
to UMA.
2005-07-14 11:52:06 +00:00
|
|
|
"Return malloc types");
|
|
|
|
|
|
|
|
|
|
SYSCTL_INT(_kern, OID_AUTO, malloc_count, CTLFLAG_RD, &kmemcount, 0,
|
|
|
|
|
"Count of kernel malloc types");
|
|
|
|
|
|
2008-05-23 00:43:36 +00:00
|
|
|
void
|
|
|
|
|
malloc_type_list(malloc_type_list_func_t *func, void *arg)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type *mtp, **bufmtp;
|
|
|
|
|
int count, i;
|
|
|
|
|
size_t buflen;
|
|
|
|
|
|
|
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
restart:
|
|
|
|
|
mtx_assert(&malloc_mtx, MA_OWNED);
|
|
|
|
|
count = kmemcount;
|
|
|
|
|
mtx_unlock(&malloc_mtx);
|
|
|
|
|
|
|
|
|
|
buflen = sizeof(struct malloc_type *) * count;
|
|
|
|
|
bufmtp = malloc(buflen, M_TEMP, M_WAITOK);
|
|
|
|
|
|
|
|
|
|
mtx_lock(&malloc_mtx);
|
|
|
|
|
|
|
|
|
|
if (count < kmemcount) {
|
|
|
|
|
free(bufmtp, M_TEMP);
|
|
|
|
|
goto restart;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
for (mtp = kmemstatistics, i = 0; mtp != NULL; mtp = mtp->ks_next, i++)
|
|
|
|
|
bufmtp[i] = mtp;
|
|
|
|
|
|
|
|
|
|
mtx_unlock(&malloc_mtx);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < count; i++)
|
|
|
|
|
(func)(bufmtp[i], arg);
|
|
|
|
|
|
|
|
|
|
free(bufmtp, M_TEMP);
|
|
|
|
|
}
|
|
|
|
|
|
2005-10-20 17:41:47 +00:00
|
|
|
#ifdef DDB
|
2019-10-11 01:31:31 +00:00
|
|
|
static int64_t
|
|
|
|
|
get_malloc_stats(const struct malloc_type_internal *mtip, uint64_t *allocs,
|
|
|
|
|
uint64_t *inuse)
|
|
|
|
|
{
|
|
|
|
|
const struct malloc_type_stats *mtsp;
|
|
|
|
|
uint64_t frees, alloced, freed;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
*allocs = 0;
|
|
|
|
|
frees = 0;
|
|
|
|
|
alloced = 0;
|
|
|
|
|
freed = 0;
|
|
|
|
|
for (i = 0; i <= mp_maxid; i++) {
|
|
|
|
|
mtsp = zpcpu_get_cpu(mtip->mti_stats, i);
|
|
|
|
|
|
|
|
|
|
*allocs += mtsp->mts_numallocs;
|
|
|
|
|
frees += mtsp->mts_numfrees;
|
|
|
|
|
alloced += mtsp->mts_memalloced;
|
|
|
|
|
freed += mtsp->mts_memfreed;
|
|
|
|
|
}
|
|
|
|
|
*inuse = *allocs - frees;
|
|
|
|
|
return (alloced - freed);
|
|
|
|
|
}
|
|
|
|
|
|
2005-10-20 17:41:47 +00:00
|
|
|
DB_SHOW_COMMAND(malloc, db_show_malloc)
|
|
|
|
|
{
|
2019-10-11 01:31:31 +00:00
|
|
|
const char *fmt_hdr, *fmt_entry;
|
2005-10-20 17:41:47 +00:00
|
|
|
struct malloc_type *mtp;
|
2019-10-11 01:31:31 +00:00
|
|
|
uint64_t allocs, inuse;
|
|
|
|
|
int64_t size;
|
|
|
|
|
/* variables for sorting */
|
|
|
|
|
struct malloc_type *last_mtype, *cur_mtype;
|
|
|
|
|
int64_t cur_size, last_size;
|
|
|
|
|
int ties;
|
|
|
|
|
|
|
|
|
|
if (modif[0] == 'i') {
|
|
|
|
|
fmt_hdr = "%s,%s,%s,%s\n";
|
|
|
|
|
fmt_entry = "\"%s\",%ju,%jdK,%ju\n";
|
|
|
|
|
} else {
|
|
|
|
|
fmt_hdr = "%18s %12s %12s %12s\n";
|
|
|
|
|
fmt_entry = "%18s %12ju %12jdK %12ju\n";
|
|
|
|
|
}
|
2005-10-20 17:41:47 +00:00
|
|
|
|
2019-10-11 01:31:31 +00:00
|
|
|
db_printf(fmt_hdr, "Type", "InUse", "MemUse", "Requests");
|
|
|
|
|
|
|
|
|
|
/* Select sort, largest size first. */
|
|
|
|
|
last_mtype = NULL;
|
|
|
|
|
last_size = INT64_MAX;
|
|
|
|
|
for (;;) {
|
|
|
|
|
cur_mtype = NULL;
|
|
|
|
|
cur_size = -1;
|
|
|
|
|
ties = 0;
|
|
|
|
|
|
|
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
/*
|
|
|
|
|
* In the case of size ties, print out mtypes
|
|
|
|
|
* in the order they are encountered. That is,
|
|
|
|
|
* when we encounter the most recently output
|
|
|
|
|
* mtype, we have already printed all preceding
|
|
|
|
|
* ties, and we must print all following ties.
|
|
|
|
|
*/
|
|
|
|
|
if (mtp == last_mtype) {
|
|
|
|
|
ties = 1;
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
size = get_malloc_stats(mtp->ks_handle, &allocs,
|
|
|
|
|
&inuse);
|
|
|
|
|
if (size > cur_size && size < last_size + ties) {
|
|
|
|
|
cur_size = size;
|
|
|
|
|
cur_mtype = mtp;
|
|
|
|
|
}
|
2005-10-20 17:41:47 +00:00
|
|
|
}
|
2019-10-11 01:31:31 +00:00
|
|
|
if (cur_mtype == NULL)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
size = get_malloc_stats(cur_mtype->ks_handle, &allocs, &inuse);
|
|
|
|
|
db_printf(fmt_entry, cur_mtype->ks_shortdesc, inuse,
|
|
|
|
|
howmany(size, 1024), allocs);
|
|
|
|
|
|
2012-07-02 16:14:52 +00:00
|
|
|
if (db_pager_quit)
|
|
|
|
|
break;
|
2019-10-11 01:31:31 +00:00
|
|
|
|
|
|
|
|
last_mtype = cur_mtype;
|
|
|
|
|
last_size = cur_size;
|
2005-10-20 17:41:47 +00:00
|
|
|
}
|
|
|
|
|
}
|
2010-07-28 15:36:12 +00:00
|
|
|
|
|
|
|
|
#if MALLOC_DEBUG_MAXZONES > 1
|
|
|
|
|
DB_SHOW_COMMAND(multizone_matches, db_show_multizone_matches)
|
|
|
|
|
{
|
|
|
|
|
struct malloc_type_internal *mtip;
|
|
|
|
|
struct malloc_type *mtp;
|
|
|
|
|
u_int subzone;
|
|
|
|
|
|
|
|
|
|
if (!have_addr) {
|
|
|
|
|
db_printf("Usage: show multizone_matches <malloc type/addr>\n");
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
mtp = (void *)addr;
|
|
|
|
|
if (mtp->ks_magic != M_MAGIC) {
|
|
|
|
|
db_printf("Magic %lx does not match expected %x\n",
|
|
|
|
|
mtp->ks_magic, M_MAGIC);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
subzone = mtip->mti_zone;
|
|
|
|
|
|
|
|
|
|
for (mtp = kmemstatistics; mtp != NULL; mtp = mtp->ks_next) {
|
|
|
|
|
mtip = mtp->ks_handle;
|
|
|
|
|
if (mtip->mti_zone != subzone)
|
|
|
|
|
continue;
|
|
|
|
|
db_printf("%s\n", mtp->ks_shortdesc);
|
2012-07-02 16:14:52 +00:00
|
|
|
if (db_pager_quit)
|
|
|
|
|
break;
|
2010-07-28 15:36:12 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
#endif /* MALLOC_DEBUG_MAXZONES > 1 */
|
|
|
|
|
#endif /* DDB */
|
2005-10-20 17:41:47 +00:00
|
|
|
|
2002-04-15 05:24:01 +00:00
|
|
|
#ifdef MALLOC_PROFILE
|
|
|
|
|
|
|
|
|
|
static int
|
|
|
|
|
sysctl_kern_mprof(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
struct sbuf sbuf;
|
2002-04-15 05:24:01 +00:00
|
|
|
uint64_t count;
|
|
|
|
|
uint64_t waste;
|
|
|
|
|
uint64_t mem;
|
|
|
|
|
int error;
|
|
|
|
|
int rsize;
|
|
|
|
|
int size;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
waste = 0;
|
|
|
|
|
mem = 0;
|
|
|
|
|
|
2011-01-27 00:34:12 +00:00
|
|
|
error = sysctl_wire_old_buffer(req, 0);
|
|
|
|
|
if (error != 0)
|
|
|
|
|
return (error);
|
2010-09-16 16:13:12 +00:00
|
|
|
sbuf_new_for_sysctl(&sbuf, NULL, 128, req);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_printf(&sbuf,
|
2002-04-15 05:24:01 +00:00
|
|
|
"\n Size Requests Real Size\n");
|
|
|
|
|
for (i = 0; i < KMEM_ZSIZE; i++) {
|
|
|
|
|
size = i << KMEM_ZSHIFT;
|
|
|
|
|
rsize = kmemzones[kmemsize[i]].kz_size;
|
|
|
|
|
count = (long long unsigned)krequests[i];
|
|
|
|
|
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_printf(&sbuf, "%6d%28llu%11d\n", size,
|
|
|
|
|
(unsigned long long)count, rsize);
|
2002-04-15 05:24:01 +00:00
|
|
|
|
|
|
|
|
if ((rsize * count) > (size * count))
|
|
|
|
|
waste += (rsize * count) - (size * count);
|
|
|
|
|
mem += (rsize * count);
|
|
|
|
|
}
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_printf(&sbuf,
|
2002-04-15 05:24:01 +00:00
|
|
|
"\nTotal memory used:\t%30llu\nTotal Memory wasted:\t%30llu\n",
|
|
|
|
|
(unsigned long long)mem, (unsigned long long)waste);
|
2010-09-16 16:13:12 +00:00
|
|
|
error = sbuf_finish(&sbuf);
|
Kernel malloc layers malloc_type allocation over one of two underlying
allocators: a set of power-of-two UMA zones for small allocations, and the
VM page allocator for large allocations. In order to maintain unified
statistics for specific malloc types, kernel malloc maintains a separate
per-type statistics pool, which can be monitored using vmstat -m. Prior
to this commit, each pool of per-type statistics was protected using a
per-type mutex associated with the malloc type.
This change modifies kernel malloc to maintain per-CPU statistics pools
for each malloc type, and protects writing those statistics using critical
sections. It also moves to unsynchronized reads of per-CPU statistics
when generating coalesced statistics. To do this, several changes are
implemented:
- In the previous world order, the statistics memory was allocated by
the owner of the malloc type structure, allocated statically using
MALLOC_DEFINE(). This embedded the definition of the malloc_type
structure into all kernel modules. Move to a model in which a pointer
within struct malloc_type points at a UMA-allocated
malloc_type_internal data structure owned and maintained by
kern_malloc.c, and not part of the exported ABI/API to the rest of
the kernel. For the purposes of easing a possible MFC, re-use an
existing pointer in 'struct malloc_type', and maintain the current
malloc_type structure size, as well as layout with respect to the
fields reused outside of the malloc subsystem (such as ks_shortdesc).
There are several unused fields as a result of no longer requiring
the mutex in malloc_type.
- Struct malloc_type_internal contains an array of malloc_type_stats,
of size MAXCPU. The structure defined above avoids hard-coding a
kernel compile-time value of MAXCPU into kernel modules that interact
with malloc.
- When accessing per-cpu statistics for a malloc type, surround read -
modify - update requests with critical_enter()/critical_exit() in
order to avoid races during write. The per-CPU fields are written
only from the CPU that owns them.
- Per-CPU stats now maintained "allocated" and "freed" counters for
number of allocations/frees and bytes allocated/freed, since there is
no longer a coherent global notion of the totals. When coalescing
malloc stats, accept a slight race between reading stats across CPUs,
and avoid showing the user a negative allocation count for the type
in the event of a race. The global high watermark is no longer
maintained for a malloc type, as there is no global notion of the
number of allocations.
- While tearing up the sysctl() path, also switch to using sbufs. The
current "export as text" sysctl format is retained with the same
syntax. We may want to change this in the future to export more
per-CPU information, such as how allocations and frees are balanced
across CPUs.
This change results in a substantial speedup of kernel malloc and free
paths on SMP, as critical sections (where usable) out-perform mutexes
due to avoiding atomic/bus-locked operations. There is also a minor
improvement on UP due to the slightly lower cost of critical sections
there. The cost of the change to this approach is the loss of a
continuous notion of total allocations that can be exploited to track
per-type high watermarks, as well as increased complexity when
monitoring statistics.
Due to carefully avoiding changing the ABI, as well as hardening the ABI
against future changes, it is not necessary to recompile kernel modules
for this change. However, MFC'ing this change to RELENG_5 will require
also MFC'ing optimizations for soft critical sections, which may modify
exposed kernel ABIs. The internal malloc API is changed, and
modifications to vmstat in order to restore "vmstat -m" on core dumps will
follow shortly.
Several improvements from: bde
Statistics approach discussed with: ups
Tested by: scottl, others
2005-05-29 13:38:07 +00:00
|
|
|
sbuf_delete(&sbuf);
|
2002-04-15 05:24:01 +00:00
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2020-02-26 14:26:36 +00:00
|
|
|
SYSCTL_OID(_kern, OID_AUTO, mprof,
|
|
|
|
|
CTLTYPE_STRING | CTLFLAG_RD | CTLFLAG_NEEDGIANT, NULL, 0,
|
|
|
|
|
sysctl_kern_mprof, "A",
|
|
|
|
|
"Malloc Profiling");
|
2002-04-15 05:24:01 +00:00
|
|
|
#endif /* MALLOC_PROFILE */
|
