d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
747bef5b85
freebsd-nq/lib/libc/stdlib/malloc.c

4014 lines
101 KiB
C
Raw Normal View History

Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*-
* Copyright (C) 2006 Jason Evans <jasone@FreeBSD.org>.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice(s), this list of conditions and the following disclaimer as
* the first lines of this file unmodified other than the possible
* addition of one or more copyright notices.
* 2. Redistributions in binary form must reproduce the above copyright
* notice(s), this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
* BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
* EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*******************************************************************************
*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* This allocator implementation is designed to provide scalable performance
* for multi-threaded programs on multi-processor systems. The following
* features are included for this purpose:
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
* + Multiple arenas are used if there are multiple CPUs, which reduces lock
* contention and cache sloshing.
*
* + Cache line sharing between arenas is avoided for internal data
* structures.
*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* + Memory is managed in chunks and runs (chunks can be split into runs using
* a binary buddy scheme), rather than as individual pages. This provides
* a constant-time mechanism for associating allocations with particular
* arenas.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* Allocation requests are rounded up to the nearest size class, and no record
* of the original request size is maintained. Allocations are broken into
* categories according to size class. Assuming runtime defaults, 4 kB pages
* and a 16 byte quantum, the size classes in each category are as follows:
*
* |====================================|
* | Category | Subcategory | Size |
* |====================================|
* | Small | Tiny | 2 |
* | | | 4 |
* | | | 8 |
* | |----------------+--------|
* | | Quantum-spaced | 16 |
* | | | 32 |
* | | | 48 |
* | | | ... |
* | | | 480 |
* | | | 496 |
* | | | 512 |
* | |----------------+--------|
* | | Sub-page | 1 kB |
* | | | 2 kB |
* |====================================|
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
* | Large | 4 kB |
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* | | 8 kB |
* | | 16 kB |
* | | ... |
* | | 256 kB |
* | | 512 kB |
* |====================================|
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* | Huge | 1 MB |
* | | 2 MB |
* | | 3 MB |
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* | | ... |
* |====================================|
*
* A different mechanism is used for each category:
*
* Small : Each size class is segregated into its own set of runs. Each run
* maintains a bitmap of which regions are free/allocated.
*
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
* Large : Each allocation is backed by a dedicated run. Metadata are stored
* in the associated arena chunk header maps.
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
*
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
* Huge : Each allocation is backed by a dedicated contiguous set of chunks.
* Metadata are stored in a separate red-black tree.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
*******************************************************************************
*/
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*******************************************************************************
*
* Ring macros.
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*******************************************************************************
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Ring definitions. */
#define qr(a_type) struct { \
a_type *qre_next; \
a_type *qre_prev; \
}
#define qr_initializer {NULL, NULL}
/* Ring functions. */
#define qr_new(a_qr, a_field) do { \
(a_qr)->a_field.qre_next = (a_qr); \
(a_qr)->a_field.qre_prev = (a_qr); \
} while (0)
#define qr_next(a_qr, a_field) ((a_qr)->a_field.qre_next)
#define qr_prev(a_qr, a_field) ((a_qr)->a_field.qre_prev)
#define qr_before_insert(a_qrelm, a_qr, a_field) do { \
(a_qr)->a_field.qre_prev = (a_qrelm)->a_field.qre_prev; \
(a_qr)->a_field.qre_next = (a_qrelm); \
(a_qr)->a_field.qre_prev->a_field.qre_next = (a_qr); \
(a_qrelm)->a_field.qre_prev = (a_qr); \
} while (0)
#define qr_after_insert(a_qrelm, a_qr, a_field) do { \
(a_qr)->a_field.qre_next = (a_qrelm)->a_field.qre_next; \
(a_qr)->a_field.qre_prev = (a_qrelm); \
(a_qr)->a_field.qre_next->a_field.qre_prev = (a_qr); \
(a_qrelm)->a_field.qre_next = (a_qr); \
} while (0)
#define qr_meld(a_qr_a, a_qr_b, a_type, a_field) do { \
a_type *t; \
(a_qr_a)->a_field.qre_prev->a_field.qre_next = (a_qr_b); \
(a_qr_b)->a_field.qre_prev->a_field.qre_next = (a_qr_a); \
t = (a_qr_a)->a_field.qre_prev; \
(a_qr_a)->a_field.qre_prev = (a_qr_b)->a_field.qre_prev; \
(a_qr_b)->a_field.qre_prev = t; \
} while (0)
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/*
* qr_meld() and qr_split() are functionally equivalent, so there's no need to
* have two copies of the code.
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#define qr_split(a_qr_a, a_qr_b, a_type, a_field) \
qr_meld((a_qr_a), (a_qr_b), a_type, a_field)
#define qr_remove(a_qr, a_field) do { \
(a_qr)->a_field.qre_prev->a_field.qre_next \
= (a_qr)->a_field.qre_next; \
(a_qr)->a_field.qre_next->a_field.qre_prev \
= (a_qr)->a_field.qre_prev; \
(a_qr)->a_field.qre_next = (a_qr); \
(a_qr)->a_field.qre_prev = (a_qr); \
} while (0)
#define qr_foreach(var, a_qr, a_field) \
for ((var) = (a_qr); \
(var) != NULL; \
(var) = (((var)->a_field.qre_next != (a_qr)) \
? (var)->a_field.qre_next : NULL))
#define qr_reverse_foreach(var, a_qr, a_field) \
for ((var) = ((a_qr) != NULL) ? qr_prev(a_qr, a_field) : NULL; \
(var) != NULL; \
(var) = (((var) != (a_qr)) \
? (var)->a_field.qre_prev : NULL))
/******************************************************************************/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* MALLOC_PRODUCTION disables assertions and statistics gathering. It also
* defaults the A and J runtime options to off. These settings are appropriate
* for production systems.
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
*/
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* #define MALLOC_PRODUCTION */
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifndef MALLOC_PRODUCTION
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
# define MALLOC_DEBUG
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix the style of the SCM ID's. I believe have made all of libc .c's as consistent as possible.
2002-03-22 21:53:29 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#include "libc_private.h"
#ifdef MALLOC_DEBUG
# define _LOCK_DEBUG
#endif
#include "spinlock.h"
#include "namespace.h"
#include <sys/mman.h>
#include <sys/param.h>
#include <sys/stddef.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/sysctl.h>
#include <sys/tree.h>
#include <sys/uio.h>
#include <sys/ktrace.h> /* Must come after several other sys/ includes. */
#include <machine/atomic.h>
#include <machine/cpufunc.h>
#include <machine/vmparam.h>
#include <errno.h>
#include <limits.h>
#include <pthread.h>
#include <sched.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdio.h>
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#include <stdint.h>
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <unistd.h>
#include "un-namespace.h"
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* MALLOC_STATS enables statistics calculation. */
#ifndef MALLOC_PRODUCTION
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
# define MALLOC_STATS
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifndef MALLOC_DEBUG
# ifndef NDEBUG
# define NDEBUG
# endif
#endif
#include <assert.h>
#ifdef MALLOC_DEBUG
/* Disable inlining to make debugging easier. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
# define inline
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
/* Size of stack-allocated buffer passed to strerror_r(). */
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define STRERROR_BUF 64
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Minimum alignment of allocations is 2^QUANTUM_2POW_MIN bytes. */
#ifdef __i386__
# define QUANTUM_2POW_MIN 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 2
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define USE_BRK
#endif
#ifdef __ia64__
# define QUANTUM_2POW_MIN 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef __alpha__
# define QUANTUM_2POW_MIN 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define NO_TLS
#endif
#ifdef __sparc64__
# define QUANTUM_2POW_MIN 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define NO_TLS
#endif
#ifdef __amd64__
# define QUANTUM_2POW_MIN 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef __arm__
Conditionally expand the size_invs lookup table in arena_run_reg_dalloc() so that architectures with a quantum of 8 (rather than 16) work. Restore arm's quantum to 8. Submitted by: jmg
2006-07-27 19:09:32 +00:00
# define QUANTUM_2POW_MIN 3
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 2
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define USE_BRK
# define NO_TLS
#endif
#ifdef __powerpc__
# define QUANTUM_2POW_MIN 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 2
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define USE_BRK
#endif
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define SIZEOF_PTR (1 << SIZEOF_PTR_2POW)
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
/* sizeof(int) == (1 << SIZEOF_INT_2POW). */
#ifndef SIZEOF_INT_2POW
# define SIZEOF_INT_2POW 2
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* We can't use TLS in non-PIC programs, since TLS relies on loader magic. */
#if (!defined(PIC) && !defined(NO_TLS))
# define NO_TLS
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Size and alignment of memory chunks that are allocated by the OS's virtual
* memory system.
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define CHUNK_2POW_DEFAULT 20
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Maximum size of L1 cache line. This is used to avoid cache line aliasing,
* so over-estimates are okay (up to a point), but under-estimates will
* negatively affect performance.
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define CACHELINE_2POW 6
#define CACHELINE ((size_t)(1 << CACHELINE_2POW))
/* Smallest size class to support. */
#define TINY_MIN_2POW 1
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
* Maximum size class that is a multiple of the quantum, but not (necessarily)
* a power of 2. Above this size, allocations are rounded up to the nearest
* power of 2.
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define SMALL_MAX_2POW_DEFAULT 9
#define SMALL_MAX_DEFAULT (1 << SMALL_MAX_2POW_DEFAULT)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
* Maximum desired run header overhead. Runs are sized as small as possible
* such that this setting is still honored, without violating other constraints.
* The goal is to make runs as small as possible without exceeding a per run
* external fragmentation threshold.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
* Note that it is possible to set this low enough that it cannot be honored
* for some/all object sizes, since there is one bit of header overhead per
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* object (plus a constant). In such cases, this constraint is relaxed.
*
* RUN_MAX_OVRHD_RELAX specifies the maximum number of bits per region of
* overhead for which RUN_MAX_OVRHD is relaxed.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*/
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#define RUN_MAX_OVRHD 0.015
#define RUN_MAX_OVRHD_RELAX 1.5
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Put a cap on small object run size. This overrides RUN_MAX_OVRHD. */
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define RUN_MAX_SMALL_2POW 16
#define RUN_MAX_SMALL (1 << RUN_MAX_SMALL_2POW)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
Malloc flag X makes malloc behave like the canonical xmalloc() wrapper. Untested support for Solaris from John-Mark Gurney Reviewed by: phk Submitted by: (partially) John-Mark Gurney <gurney_j@resnet.uoregon.edu>
1997-05-30 20:39:32 +00:00
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Mutexes based on spinlocks. We can't use normal pthread mutexes, because
* they require malloc()ed memory.
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
typedef struct {
spinlock_t lock;
} malloc_mutex_t;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
/* Set to true once the allocator has been initialized. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static bool malloc_initialized = false;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
/* Used to avoid initialization races. */
static malloc_mutex_t init_lock = {_SPINLOCK_INITIALIZER};
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
/*
* Statistics data structures.
*/
#ifdef MALLOC_STATS
typedef struct malloc_bin_stats_s malloc_bin_stats_t;
struct malloc_bin_stats_s {
/*
* Number of allocation requests that corresponded to the size of this
* bin.
*/
uint64_t nrequests;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Total number of runs created for this bin's size class. */
uint64_t nruns;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Total number of run promotions/demotions for this bin's size class.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
uint64_t npromote;
uint64_t ndemote;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* High-water mark for this bin. */
unsigned long highruns;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Current number of runs in this bin. */
unsigned long curruns;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
typedef struct arena_stats_s arena_stats_t;
struct arena_stats_s {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Number of bytes currently mapped. */
size_t mapped;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Per-size-category statistics. */
size_t allocated_small;
uint64_t nmalloc_small;
uint64_t ndalloc_small;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
size_t allocated_large;
uint64_t nmalloc_large;
uint64_t ndalloc_large;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
typedef struct chunk_stats_s chunk_stats_t;
struct chunk_stats_s {
/* Number of chunks that were allocated. */
uint64_t nchunks;
/* High-water mark for number of chunks allocated. */
unsigned long highchunks;
/*
* Current number of chunks allocated. This value isn't maintained for
* any other purpose, so keep track of it in order to be able to set
* highchunks.
*/
unsigned long curchunks;
};
#endif /* #ifdef MALLOC_STATS */
/******************************************************************************/
/*
* Chunk data structures.
*/
/* Tree of chunks. */
typedef struct chunk_node_s chunk_node_t;
struct chunk_node_s {
/* Linkage for the chunk tree. */
RB_ENTRY(chunk_node_s) link;
/*
* Pointer to the chunk that this tree node is responsible for. In some
* (but certainly not all) cases, this data structure is placed at the
* beginning of the corresponding chunk, so this field may point to this
* node.
*/
void *chunk;
/* Total chunk size. */
size_t size;
};
typedef struct chunk_tree_s chunk_tree_t;
RB_HEAD(chunk_tree_s, chunk_node_s);
/******************************************************************************/
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Arena data structures.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
typedef struct arena_s arena_t;
typedef struct arena_bin_s arena_bin_t;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
typedef struct arena_chunk_map_s arena_chunk_map_t;
struct arena_chunk_map_s {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Number of pages in run. */
uint32_t npages;
/*
* Position within run. For a free run, this is POS_FREE for the first
* and last pages. The POS_FREE special value makes it possible to
* quickly coalesce free runs.
*
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
* This is the limiting factor for chunksize; there can be at most 2^31
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* pages in a run.
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
#define POS_FREE ((uint32_t)0xffffffffU)
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
uint32_t pos;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Arena chunk header. */
typedef struct arena_chunk_s arena_chunk_t;
struct arena_chunk_s {
/* Arena that owns the chunk. */
arena_t *arena;
/* Linkage for the arena's chunk tree. */
RB_ENTRY(arena_chunk_s) link;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Number of pages in use. This is maintained in order to make
* detection of empty chunks fast.
*/
uint32_t pages_used;
/*
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* Every time a free run larger than this value is created/coalesced,
* this value is increased. The only way that the value decreases is if
* arena_run_alloc() fails to find a free run as large as advertised by
* this value.
*/
uint32_t max_frun_npages;
/*
* Every time a free run that starts at an earlier page than this value
* is created/coalesced, this value is decreased. It is reset in a
* similar fashion to max_frun_npages.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
uint32_t min_frun_ind;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/*
* Map of pages within chunk that keeps track of free/large/small. For
* free runs, only the map entries for the first and last pages are
* kept up to date, so that free runs can be quickly coalesced.
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_map_t map[1]; /* Dynamically sized. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
typedef struct arena_chunk_tree_s arena_chunk_tree_t;
RB_HEAD(arena_chunk_tree_s, arena_chunk_s);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
typedef struct arena_run_s arena_run_t;
struct arena_run_s {
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
/* Linkage for run rings. */
qr(arena_run_t) link;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_DEBUG
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
uint32_t magic;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
# define ARENA_RUN_MAGIC 0x384adf93
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Bin this run is associated with. */
arena_bin_t *bin;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Index of first element that might have a free region. */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
unsigned regs_minelm;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Number of free regions in run. */
Increase the minimum chunk size by a power of two (32kB --> 64kB, assuming 4kB pages), in order to avoid dangerous rounding error when calculating fullness limits during run promotion/demotion. Convert a structure bitfield to a normal field in areana_run_t. This should have been changed along with the other fields in revision 1.120.
2006-05-10 00:07:45 +00:00
unsigned nfree;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
* Current quartile for this run, one of: {RUN_QINIT, RUN_Q0, RUN_25,
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
* RUN_Q50, RUN_Q75, RUN_Q100}.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
#define RUN_QINIT 0
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
#define RUN_Q0 1
#define RUN_Q25 2
#define RUN_Q50 3
#define RUN_Q75 4
#define RUN_Q100 5
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
unsigned quartile;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
* Limits on the number of free regions for the fullness quartile this
* run is currently in. If nfree goes outside these limits, the run
* is moved to a different fullness quartile.
*/
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
unsigned free_max;
unsigned free_min;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/* Bitmask of in-use regions (0: in use, 1: free). */
unsigned regs_mask[1]; /* Dynamically sized. */
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
};
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
/* Used for run ring headers, where the run isn't actually used. */
typedef struct arena_run_link_s arena_run_link_t;
struct arena_run_link_s {
/* Linkage for run rings. */
qr(arena_run_t) link;
};
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
/* Avoid pointer aliasing issues. */
static inline arena_run_t *
arena_bin_link(void *ptr)
{
return ((arena_run_t *)ptr);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
struct arena_bin_s {
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Current run being used to service allocations of this bin's size
* class.
*/
arena_run_t *runcur;
/*
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
* Links into rings of runs, of various fullnesses (names indicate
* approximate lower bounds). A new run conceptually starts off in
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* runsinit, and it isn't inserted into the runs0 ring until it
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
* reaches 25% full (hysteresis mechanism). For the run to be moved
* again, it must become either empty or 50% full. Thus, each ring
* contains runs that are within 50% above the advertised fullness for
* the ring. This provides a low-overhead mechanism for segregating
* runs into approximate fullness classes.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
* Conceptually, there is a runs100 that contains completely full runs.
* Since we don't need to search for these runs though, no runs100 ring
* is actually maintained.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
* These rings are useful when looking for an existing run to use when
* runcur is no longer usable. We look for usable runs in the
* following order:
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
* 1) runs50
* 2) runs25
* 3) runs0
* 4) runs75
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* runs75 isn't a good place to look, because it contains runs that may
* be nearly completely full. Still, we look there as a last resort in
* order to avoid allocating a new run if at all possible.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* arena_run_link_t runsinit; 0% <= fullness < 25% */
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
arena_run_link_t runs0; /* 0% < fullness < 50% */
arena_run_link_t runs25; /* 25% < fullness < 75% */
arena_run_link_t runs50; /* 50% < fullness < 100% */
arena_run_link_t runs75; /* 75% < fullness < 100% */
/* arena_run_link_t runs100; fullness == 100% */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Size of regions in a run for this bin's size class. */
size_t reg_size;
/* Total size of a run for this bin's size class. */
size_t run_size;
/* Total number of regions in a run for this bin's size class. */
uint32_t nregs;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/* Number of elements in a run's regs_mask for this bin's size class. */
uint32_t regs_mask_nelms;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Offset of first region in a run for this bin's size class. */
uint32_t reg0_offset;
#ifdef MALLOC_STATS
/* Bin statistics. */
malloc_bin_stats_t stats;
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
struct arena_s {
#ifdef MALLOC_DEBUG
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
uint32_t magic;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define ARENA_MAGIC 0x947d3d24
#endif
/* All operations on this arena require that mtx be locked. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
malloc_mutex_t mtx;
#ifdef MALLOC_STATS
arena_stats_t stats;
#endif
/*
* Tree of chunks this arena manages.
*/
arena_chunk_tree_t chunks;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
/*
* In order to avoid rapid chunk allocation/deallocation when an arena
* oscillates right on the cusp of needing a new chunk, cache the most
* recently freed chunk. This caching is disabled by opt_hint.
*
* There is one spare chunk per arena, rather than one spare total, in
* order to avoid interactions between multiple threads that could make
* a single spare inadequate.
*/
arena_chunk_t *spare;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* bins is used to store rings of free regions of the following sizes,
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* assuming a 16-byte quantum, 4kB pagesize, and default MALLOC_OPTIONS.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
* bins[i] | size |
* --------+------+
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* 0 | 2 |
* 1 | 4 |
* 2 | 8 |
* --------+------+
* 3 | 16 |
* 4 | 32 |
* 5 | 48 |
* 6 | 64 |
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* : :
* : :
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* 33 | 496 |
* 34 | 512 |
* --------+------+
* 35 | 1024 |
* 36 | 2048 |
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* --------+------+
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_bin_t bins[1]; /* Dynamically sized. */
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
};
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
/*
* Data.
*/
/* Number of CPUs. */
static unsigned ncpus;
/* VM page size. */
static unsigned pagesize;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
static unsigned pagesize_mask;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static unsigned pagesize_2pow;
/* Various bin-related settings. */
static size_t bin_maxclass; /* Max size class for bins. */
static unsigned ntbins; /* Number of (2^n)-spaced tiny bins. */
static unsigned nqbins; /* Number of quantum-spaced bins. */
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
static unsigned nsbins; /* Number of (2^n)-spaced sub-page bins. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static size_t small_min;
static size_t small_max;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Various quantum-related settings. */
static size_t quantum;
static size_t quantum_mask; /* (quantum - 1). */
/* Various chunk-related settings. */
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
static size_t chunksize;
static size_t chunksize_mask; /* (chunksize - 1). */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
static unsigned chunk_npages;
static unsigned arena_chunk_header_npages;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static size_t arena_maxclass; /* Max size class for arenas. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/********/
/*
* Chunks.
*/
/* Protects chunk-related data structures. */
static malloc_mutex_t chunks_mtx;
/* Tree of chunks that are stand-alone huge allocations. */
static chunk_tree_t huge;
#ifdef USE_BRK
/*
* Try to use brk for chunk-size allocations, due to address space constraints.
*/
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
/*
* Protects sbrk() calls. This must be separate from chunks_mtx, since
* base_chunk_alloc() also uses sbrk(), but cannot lock chunks_mtx (doing so
* could cause recursive lock acquisition).
*/
static malloc_mutex_t brk_mtx;
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
/* Result of first sbrk(0) call. */
static void *brk_base;
/* Current end of brk, or ((void *)-1) if brk is exhausted. */
static void *brk_prev;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
/* Current upper limit on brk addresses. */
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
static void *brk_max;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef MALLOC_STATS
/*
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* Byte counters for allocated/mapped space used by the chunks in the huge
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* allocations tree.
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static uint64_t huge_nmalloc;
static uint64_t huge_ndalloc;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static size_t huge_allocated;
#endif
/*
* Tree of chunks that were previously allocated. This is used when allocating
* chunks, in an attempt to re-use address space.
*/
static chunk_tree_t old_chunks;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/****************************/
/*
* base (internal allocation).
*/
/*
* Current chunk that is being used for internal memory allocations. This
* chunk is carved up in cacheline-size quanta, so that there is no chance of
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* false cache line sharing.
*/
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
static void *base_chunk;
static void *base_next_addr;
static void *base_past_addr; /* Addr immediately past base_chunk. */
static chunk_node_t *base_chunk_nodes; /* LIFO cache of chunk nodes. */
static malloc_mutex_t base_mtx;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
static size_t base_mapped;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/********/
/*
* Arenas.
*/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Arenas that are used to service external requests. Not all elements of the
* arenas array are necessarily used; arenas are created lazily as needed.
*/
static arena_t **arenas;
static unsigned narenas;
#ifndef NO_TLS
static unsigned next_arena;
#endif
static malloc_mutex_t arenas_mtx; /* Protects arenas initialization. */
#ifndef NO_TLS
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Map of pthread_self() --> arenas[???], used for selecting an arena to use
* for allocations.
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
static __thread arena_t *arenas_map;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef MALLOC_STATS
/* Chunk statistics. */
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
static chunk_stats_t stats_chunks;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
/*******************************/
/*
* Runtime configuration options.
*/
const char *_malloc_options;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifndef MALLOC_PRODUCTION
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static bool opt_abort = true;
static bool opt_junk = true;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#else
static bool opt_abort = false;
static bool opt_junk = false;
#endif
static bool opt_hint = false;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static bool opt_print_stats = false;
static size_t opt_quantum_2pow = QUANTUM_2POW_MIN;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static size_t opt_small_max_2pow = SMALL_MAX_2POW_DEFAULT;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static size_t opt_chunk_2pow = CHUNK_2POW_DEFAULT;
static bool opt_utrace = false;
static bool opt_sysv = false;
static bool opt_xmalloc = false;
static bool opt_zero = false;
static int32_t opt_narenas_lshift = 0;
typedef struct {
void *p;
size_t s;
void *r;
} malloc_utrace_t;
#define UTRACE(a, b, c) \
if (opt_utrace) { \
malloc_utrace_t ut = {a, b, c}; \
utrace(&ut, sizeof(ut)); \
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Begin function prototypes for non-inline static functions.
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
*/
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void malloc_mutex_init(malloc_mutex_t *a_mutex);
static void wrtmessage(const char *p1, const char *p2, const char *p3,
const char *p4);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#ifdef MALLOC_STATS
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void malloc_printf(const char *format, ...);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#endif
static char *umax2s(uintmax_t x, char *s);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
static bool base_chunk_alloc(size_t minsize);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static void *base_alloc(size_t size);
static chunk_node_t *base_chunk_node_alloc(void);
static void base_chunk_node_dealloc(chunk_node_t *node);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void stats_print(arena_t *arena);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
static void *pages_map(void *addr, size_t size);
static void pages_unmap(void *addr, size_t size);
static void *chunk_alloc(size_t size);
static void chunk_dealloc(void *chunk, size_t size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#ifndef NO_TLS
static arena_t *choose_arena_hard(void);
#endif
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
static void arena_run_split(arena_t *arena, arena_run_t *run, size_t size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_chunk_t *arena_chunk_alloc(arena_t *arena);
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
static void arena_chunk_dealloc(arena_t *arena, arena_chunk_t *chunk);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
static void arena_bin_run_promote(arena_t *arena, arena_bin_t *bin,
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_run_t *run);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
static void arena_bin_run_demote(arena_t *arena, arena_bin_t *bin,
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_run_t *run);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
static arena_run_t *arena_run_alloc(arena_t *arena, size_t size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void arena_run_dalloc(arena_t *arena, arena_run_t *run, size_t size);
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
static arena_run_t *arena_bin_nonfull_run_get(arena_t *arena, arena_bin_t *bin);
static void *arena_bin_malloc_hard(arena_t *arena, arena_bin_t *bin);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
static size_t arena_bin_run_size_calc(arena_bin_t *bin, size_t min_run_size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void *arena_malloc(arena_t *arena, size_t size);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
static void *arena_palloc(arena_t *arena, size_t alignment, size_t size,
size_t alloc_size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
static size_t arena_salloc(const void *ptr);
static void *arena_ralloc(void *ptr, size_t size, size_t oldsize);
static void arena_dalloc(arena_t *arena, arena_chunk_t *chunk, void *ptr);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static bool arena_new(arena_t *arena);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static arena_t *arenas_extend(unsigned ind);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
static void *huge_malloc(size_t size);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
static void *huge_palloc(size_t alignment, size_t size);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
static void *huge_ralloc(void *ptr, size_t size, size_t oldsize);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void huge_dalloc(void *ptr);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
static void *imalloc(size_t size);
static void *ipalloc(size_t alignment, size_t size);
static void *icalloc(size_t size);
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
static size_t isalloc(const void *ptr);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
static void *iralloc(void *ptr, size_t size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void idalloc(void *ptr);
static void malloc_print_stats(void);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
static bool malloc_init_hard(void);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* End function prototypes.
*/
/******************************************************************************/
/*
* Begin mutex.
*/
static void
malloc_mutex_init(malloc_mutex_t *a_mutex)
{
static const spinlock_t lock = _SPINLOCK_INITIALIZER;
a_mutex->lock = lock;
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline void
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_lock(malloc_mutex_t *a_mutex)
{
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (__isthreaded)
_SPINLOCK(&a_mutex->lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline void
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_unlock(malloc_mutex_t *a_mutex)
{
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (__isthreaded)
_SPINUNLOCK(&a_mutex->lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
/*
* End mutex.
*/
/******************************************************************************/
/*
* Begin Utility functions/macros.
*/
/* Return the chunk address for allocation address a. */
#define CHUNK_ADDR2BASE(a) \
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
((void *)((uintptr_t)(a) & ~chunksize_mask))
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Return the chunk offset of address a. */
#define CHUNK_ADDR2OFFSET(a) \
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
((size_t)((uintptr_t)(a) & chunksize_mask))
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Return the smallest chunk multiple that is >= s. */
#define CHUNK_CEILING(s) \
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
(((s) + chunksize_mask) & ~chunksize_mask)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/* Return the smallest cacheline multiple that is >= s. */
#define CACHELINE_CEILING(s) \
(((s) + (CACHELINE - 1)) & ~(CACHELINE - 1))
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Return the smallest quantum multiple that is >= a. */
#define QUANTUM_CEILING(a) \
(((a) + quantum_mask) & ~quantum_mask)
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Return the smallest pagesize multiple that is >= s. */
#define PAGE_CEILING(s) \
(((s) + pagesize_mask) & ~pagesize_mask)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Compute the smallest power of 2 that is >= x. */
static inline size_t
pow2_ceil(size_t x)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
x--;
x |= x >> 1;
x |= x >> 2;
x |= x >> 4;
x |= x >> 8;
x |= x >> 16;
#if (SIZEOF_PTR == 8)
x |= x >> 32;
#endif
x++;
return (x);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
static void
wrtmessage(const char *p1, const char *p2, const char *p3, const char *p4)
{
_write(STDERR_FILENO, p1, strlen(p1));
_write(STDERR_FILENO, p2, strlen(p2));
_write(STDERR_FILENO, p3, strlen(p3));
_write(STDERR_FILENO, p4, strlen(p4));
}
void (*_malloc_message)(const char *p1, const char *p2, const char *p3,
const char *p4) = wrtmessage;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#ifdef MALLOC_STATS
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Print to stderr in such a way as to (hopefully) avoid memory allocation.
*/
static void
malloc_printf(const char *format, ...)
{
char buf[4096];
va_list ap;
va_start(ap, format);
vsnprintf(buf, sizeof(buf), format, ap);
va_end(ap);
_malloc_message(buf, "", "", "");
}
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#endif
/*
* We don't want to depend on vsnprintf() for production builds, since that can
* cause unnecessary bloat for static binaries. umax2s() provides minimal
* integer printing functionality, so that malloc_printf() use can be limited to
* MALLOC_STATS code.
*/
#define UMAX2S_BUFSIZE 21
static char *
umax2s(uintmax_t x, char *s)
{
unsigned i;
/* Make sure UMAX2S_BUFSIZE is large enough. */
assert(sizeof(uintmax_t) <= 8);
i = UMAX2S_BUFSIZE - 1;
s[i] = '\0';
do {
i--;
s[i] = "0123456789"[x % 10];
x /= 10;
} while (x > 0);
return (&s[i]);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/******************************************************************************/
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
static bool
base_chunk_alloc(size_t minsize)
{
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert(minsize <= chunksize);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#ifdef USE_BRK
/*
* Do special brk allocation here, since the base chunk doesn't really
* need to be chunk-aligned.
*/
if (brk_prev != (void *)-1) {
void *brk_cur;
intptr_t incr;
malloc_mutex_lock(&brk_mtx);
do {
/* Get the current end of brk. */
brk_cur = sbrk(0);
/*
* Calculate how much padding is necessary to
* chunk-align the end of brk. Don't worry about
* brk_cur not being chunk-aligned though.
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
incr = (intptr_t)chunksize
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
- (intptr_t)CHUNK_ADDR2OFFSET(brk_cur);
if (incr < minsize)
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
incr += chunksize;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
brk_prev = sbrk(incr);
if (brk_prev == brk_cur) {
/* Success. */
malloc_mutex_unlock(&brk_mtx);
base_chunk = brk_cur;
base_next_addr = base_chunk;
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
base_past_addr = (void *)((uintptr_t)base_chunk
+ incr);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
base_mapped += incr;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#endif
return (false);
}
} while (brk_prev != (void *)-1);
malloc_mutex_unlock(&brk_mtx);
}
#endif
/*
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
* Don't worry about chunk alignment here, since base_chunk doesn't
* really need to be aligned.
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
base_chunk = pages_map(NULL, chunksize);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
if (base_chunk == NULL)
return (true);
base_next_addr = base_chunk;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
base_past_addr = (void *)((uintptr_t)base_chunk + chunksize);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#ifdef MALLOC_STATS
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
base_mapped += chunksize;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#endif
return (false);
}
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static void *
base_alloc(size_t size)
{
void *ret;
size_t csize;
/* Round size up to nearest multiple of the cacheline size. */
csize = CACHELINE_CEILING(size);
malloc_mutex_lock(&base_mtx);
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
/*
* Make sure there's enough space for the allocation.
* base_chunk_alloc() does not guarantee that a newly allocated chunk
* is >= size, so loop here, rather than only trying once.
*/
while ((uintptr_t)base_next_addr + csize > (uintptr_t)base_past_addr) {
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
if (base_chunk_alloc(csize)) {
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
ret = NULL;
goto RETURN;
}
}
/* Allocate. */
ret = base_next_addr;
Use uintptr_t rather than size_t when casting pointers to integers. Also, fix the few remaining casting style(9) errors that remained after the functional change. Reported by: jmallett
2006-01-20 03:11:11 +00:00
base_next_addr = (void *)((uintptr_t)base_next_addr + csize);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
RETURN:
malloc_mutex_unlock(&base_mtx);
return (ret);
}
static chunk_node_t *
base_chunk_node_alloc(void)
{
chunk_node_t *ret;
malloc_mutex_lock(&base_mtx);
if (base_chunk_nodes != NULL) {
ret = base_chunk_nodes;
base_chunk_nodes = *(chunk_node_t **)ret;
malloc_mutex_unlock(&base_mtx);
} else {
malloc_mutex_unlock(&base_mtx);
ret = (chunk_node_t *)base_alloc(sizeof(chunk_node_t));
}
return (ret);
}
static void
base_chunk_node_dealloc(chunk_node_t *node)
{
malloc_mutex_lock(&base_mtx);
*(chunk_node_t **)node = base_chunk_nodes;
base_chunk_nodes = node;
malloc_mutex_unlock(&base_mtx);
}
/******************************************************************************/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
static void
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
stats_print(arena_t *arena)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
unsigned i;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
int gap_start;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_printf(
" allocated/mapped nmalloc ndalloc\n");
malloc_printf("small: %12llu %-12s %12llu %12llu\n",
arena->stats.allocated_small, "", arena->stats.nmalloc_small,
arena->stats.ndalloc_small);
malloc_printf("large: %12llu %-12s %12llu %12llu\n",
arena->stats.allocated_large, "", arena->stats.nmalloc_large,
arena->stats.ndalloc_large);
malloc_printf("total: %12llu/%-12llu %12llu %12llu\n",
arena->stats.allocated_small + arena->stats.allocated_large,
arena->stats.mapped,
arena->stats.nmalloc_small + arena->stats.nmalloc_large,
arena->stats.ndalloc_small + arena->stats.ndalloc_large);
malloc_printf("bins: bin size regs pgs requests newruns "
"maxruns curruns promote demote\n");
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
for (i = 0, gap_start = -1; i < ntbins + nqbins + nsbins; i++) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (arena->bins[i].stats.nrequests == 0) {
if (gap_start == -1)
gap_start = i;
} else {
if (gap_start != -1) {
if (i > gap_start + 1) {
/* Gap of more than one size class. */
malloc_printf("[%u..%u]\n",
gap_start, i - 1);
} else {
/* Gap of one size class. */
malloc_printf("[%u]\n", gap_start);
}
gap_start = -1;
}
malloc_printf(
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
"%13u %1s %4u %4u %3u %9llu %7llu"
" %7lu %7lu %7llu %7llu\n",
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
i,
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
i < ntbins ? "T" : i < ntbins + nqbins ? "Q" : "S",
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena->bins[i].reg_size,
arena->bins[i].nregs,
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena->bins[i].run_size >> pagesize_2pow,
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena->bins[i].stats.nrequests,
arena->bins[i].stats.nruns,
arena->bins[i].stats.highruns,
arena->bins[i].stats.curruns,
arena->bins[i].stats.npromote,
arena->bins[i].stats.ndemote);
}
}
if (gap_start != -1) {
if (i > gap_start + 1) {
/* Gap of more than one size class. */
malloc_printf("[%u..%u]\n", gap_start, i - 1);
} else {
/* Gap of one size class. */
malloc_printf("[%u]\n", gap_start);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
#endif
/*
* End Utility functions/macros.
*/
/******************************************************************************/
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Begin chunk management functions.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline int
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
chunk_comp(chunk_node_t *a, chunk_node_t *b)
{
assert(a != NULL);
assert(b != NULL);
Use uintptr_t rather than size_t when casting pointers to integers. Also, fix the few remaining casting style(9) errors that remained after the functional change. Reported by: jmallett
2006-01-20 03:11:11 +00:00
if ((uintptr_t)a->chunk < (uintptr_t)b->chunk)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (-1);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
else if (a->chunk == b->chunk)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (0);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
else
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (1);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
/* Generate red-black tree code for chunks. */
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
RB_GENERATE_STATIC(chunk_tree_s, chunk_node_s, link, chunk_comp);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void *
pages_map(void *addr, size_t size)
{
void *ret;
/*
* We don't use MAP_FIXED here, because it can cause the *replacement*
* of existing mappings, and we only want to create new mappings.
*/
ret = mmap(addr, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON,
-1, 0);
assert(ret != NULL);
if (ret == MAP_FAILED)
ret = NULL;
else if (addr != NULL && ret != addr) {
/*
* We succeeded in mapping memory, but not in the right place.
*/
if (munmap(ret, size) == -1) {
char buf[STRERROR_BUF];
strerror_r(errno, buf, sizeof(buf));
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in munmap(): ", buf, "\n");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (opt_abort)
abort();
}
ret = NULL;
}
assert(ret == NULL || (addr == NULL && ret != addr)
|| (addr != NULL && ret == addr));
return (ret);
}
static void
pages_unmap(void *addr, size_t size)
{
if (munmap(addr, size) == -1) {
char buf[STRERROR_BUF];
strerror_r(errno, buf, sizeof(buf));
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in munmap(): ", buf, "\n");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (opt_abort)
abort();
}
}
static void *
chunk_alloc(size_t size)
{
void *ret, *chunk;
chunk_node_t *tchunk, *delchunk;
assert(size != 0);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert((size & chunksize_mask) == 0);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_lock(&chunks_mtx);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (size == chunksize) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Check for address ranges that were previously chunks and try
* to use them.
*/
tchunk = RB_MIN(chunk_tree_s, &old_chunks);
while (tchunk != NULL) {
/* Found an address range. Try to recycle it. */
chunk = tchunk->chunk;
delchunk = tchunk;
tchunk = RB_NEXT(chunk_tree_s, &old_chunks, delchunk);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Remove delchunk from the tree. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
RB_REMOVE(chunk_tree_s, &old_chunks, delchunk);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
base_chunk_node_dealloc(delchunk);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef USE_BRK
Use uintptr_t rather than size_t when casting pointers to integers. Also, fix the few remaining casting style(9) errors that remained after the functional change. Reported by: jmallett
2006-01-20 03:11:11 +00:00
if ((uintptr_t)chunk >= (uintptr_t)brk_base
&& (uintptr_t)chunk < (uintptr_t)brk_max) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Re-use a previously freed brk chunk. */
ret = chunk;
goto RETURN;
}
#endif
if ((ret = pages_map(chunk, size)) != NULL) {
/* Success. */
goto RETURN;
}
}
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
/*
* Try to over-allocate, but allow the OS to place the allocation
* anywhere. Beware of size_t wrap-around.
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (size + chunksize > size) {
if ((ret = pages_map(NULL, size + chunksize)) != NULL) {
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
size_t offset = CHUNK_ADDR2OFFSET(ret);
/*
* Success. Clean up unneeded leading/trailing space.
*/
if (offset != 0) {
/* Leading space. */
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
pages_unmap(ret, chunksize - offset);
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
ret = (void *)((uintptr_t)ret + (chunksize -
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
offset));
/* Trailing space. */
pages_unmap((void *)((uintptr_t)ret + size),
offset);
} else {
/* Trailing space only. */
pages_unmap((void *)((uintptr_t)ret + size),
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunksize);
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
}
goto RETURN;
}
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef USE_BRK
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
/*
* Try to create allocations in brk, in order to make full use of
* limited address space.
*/
if (brk_prev != (void *)-1) {
void *brk_cur;
intptr_t incr;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
* The loop is necessary to recover from races with other
* threads that are using brk for something other than malloc.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
malloc_mutex_lock(&brk_mtx);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
do {
/* Get the current end of brk. */
brk_cur = sbrk(0);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
* Calculate how much padding is necessary to
* chunk-align the end of brk.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
incr = (intptr_t)size
- (intptr_t)CHUNK_ADDR2OFFSET(brk_cur);
if (incr == size) {
ret = brk_cur;
} else {
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
ret = (void *)((intptr_t)brk_cur + incr);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
incr += size;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
brk_prev = sbrk(incr);
if (brk_prev == brk_cur) {
/* Success. */
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
malloc_mutex_unlock(&brk_mtx);
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
brk_max = (void *)((intptr_t)ret + size);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
goto RETURN;
}
} while (brk_prev != (void *)-1);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
malloc_mutex_unlock(&brk_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* All strategies for allocation failed. */
ret = NULL;
RETURN:
#ifdef MALLOC_STATS
if (ret != NULL) {
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
stats_chunks.nchunks += (size / chunksize);
stats_chunks.curchunks += (size / chunksize);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
if (stats_chunks.curchunks > stats_chunks.highchunks)
stats_chunks.highchunks = stats_chunks.curchunks;
#endif
malloc_mutex_unlock(&chunks_mtx);
assert(CHUNK_ADDR2BASE(ret) == ret);
return (ret);
}
static void
chunk_dealloc(void *chunk, size_t size)
{
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
size_t offset;
Add a missing case for the switch statement in arena_run_reg_dalloc(). [1] Fix a leak in chunk_dealloc(). [2] Reported by: [1] djam8193ah@hotmail.com, [2] Ville-Pertti Keinonen <will@exomi.com>
2006-06-20 20:38:25 +00:00
chunk_node_t key;
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
chunk_node_t *node;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(chunk != NULL);
assert(CHUNK_ADDR2BASE(chunk) == chunk);
assert(size != 0);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert((size & chunksize_mask) == 0);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
malloc_mutex_lock(&chunks_mtx);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
#ifdef USE_BRK
if ((uintptr_t)chunk >= (uintptr_t)brk_base
&& (uintptr_t)chunk < (uintptr_t)brk_max) {
void *brk_cur;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
malloc_mutex_lock(&brk_mtx);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
/* Get the current end of brk. */
brk_cur = sbrk(0);
/*
* Try to shrink the data segment if this chunk is at the end
* of the data segment. The sbrk() call here is subject to a
* race condition with threads that use brk(2) or sbrk(2)
* directly, but the alternative would be to leak memory for
* the sake of poorly designed multi-threaded programs.
*/
if (brk_cur == brk_max
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
&& (void *)((uintptr_t)chunk + size) == brk_max
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
&& sbrk(-(intptr_t)size) == brk_max) {
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
malloc_mutex_unlock(&brk_mtx);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
if (brk_prev == brk_max) {
/* Success. */
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
brk_prev = (void *)((intptr_t)brk_max
- (intptr_t)size);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
brk_max = brk_prev;
}
goto RETURN;
} else
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
malloc_mutex_unlock(&brk_mtx);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
madvise(chunk, size, MADV_FREE);
} else
#endif
pages_unmap(chunk, size);
/*
* Iteratively create records of each chunk-sized memory region that
* 'chunk' is comprised of, so that the address range can be recycled
* if memory usage increases later on.
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
for (offset = 0; offset < size; offset += chunksize) {
Add a missing case for the switch statement in arena_run_reg_dalloc(). [1] Fix a leak in chunk_dealloc(). [2] Reported by: [1] djam8193ah@hotmail.com, [2] Ville-Pertti Keinonen <will@exomi.com>
2006-06-20 20:38:25 +00:00
/*
* It is possible for chunk to overlap existing entries in
* old_chunks if it is a huge allocation, so take care to not
* leak tree nodes.
*/
key.chunk = (void *)((uintptr_t)chunk + (uintptr_t)offset);
if (RB_FIND(chunk_tree_s, &old_chunks, &key) == NULL) {
node = base_chunk_node_alloc();
if (node == NULL)
break;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add a missing case for the switch statement in arena_run_reg_dalloc(). [1] Fix a leak in chunk_dealloc(). [2] Reported by: [1] djam8193ah@hotmail.com, [2] Ville-Pertti Keinonen <will@exomi.com>
2006-06-20 20:38:25 +00:00
node->chunk = key.chunk;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
node->size = chunksize;
Add a missing case for the switch statement in arena_run_reg_dalloc(). [1] Fix a leak in chunk_dealloc(). [2] Reported by: [1] djam8193ah@hotmail.com, [2] Ville-Pertti Keinonen <will@exomi.com>
2006-06-20 20:38:25 +00:00
RB_INSERT(chunk_tree_s, &old_chunks, node);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
#ifdef USE_BRK
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
RETURN:
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef MALLOC_STATS
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
stats_chunks.curchunks -= (size / chunksize);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
malloc_mutex_unlock(&chunks_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* End chunk management functions.
*/
/******************************************************************************/
/*
* Begin arena.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
/*
* Choose an arena based on a per-thread value (fast-path code, calls slow-path
* code if necessary.
*/
static inline arena_t *
choose_arena(void)
{
arena_t *ret;
/*
* We can only use TLS if this is a PIC library, since for the static
* library version, libc's malloc is used by TLS allocation, which
* introduces a bootstrapping issue.
*/
#ifndef NO_TLS
if (__isthreaded == false) {
/*
* Avoid the overhead of TLS for single-threaded operation. If the
* app switches to threaded mode, the initial thread may end up
* being assigned to some other arena, but this one-time switch
* shouldn't cause significant issues.
* */
return (arenas[0]);
}
ret = arenas_map;
if (ret == NULL)
ret = choose_arena_hard();
#else
if (__isthreaded) {
unsigned long ind;
/*
* Hash _pthread_self() to one of the arenas. There is a prime
* number of arenas, so this has a reasonable chance of
* working. Even so, the hashing can be easily thwarted by
* inconvenient _pthread_self() values. Without specific
* knowledge of how _pthread_self() calculates values, we can't
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
* easily do much better than this.
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
*/
ind = (unsigned long) _pthread_self() % narenas;
/*
* Optimistially assume that arenas[ind] has been initialized.
* At worst, we find out that some other thread has already
* done so, after acquiring the lock in preparation. Note that
* this lazy locking also has the effect of lazily forcing
* cache coherency; without the lock acquisition, there's no
* guarantee that modification of arenas[ind] by another thread
* would be seen on this CPU for an arbitrary amount of time.
*
* In general, this approach to modifying a synchronized value
* isn't a good idea, but in this case we only ever modify the
* value once, so things work out well.
*/
ret = arenas[ind];
if (ret == NULL) {
/*
* Avoid races with another thread that may have already
* initialized arenas[ind].
*/
malloc_mutex_lock(&arenas_mtx);
if (arenas[ind] == NULL)
ret = arenas_extend((unsigned)ind);
else
ret = arenas[ind];
malloc_mutex_unlock(&arenas_mtx);
}
} else
ret = arenas[0];
#endif
assert(ret != NULL);
return (ret);
}
#ifndef NO_TLS
/*
* Choose an arena based on a per-thread value (slow-path code only, called
* only by choose_arena()).
*/
static arena_t *
choose_arena_hard(void)
{
arena_t *ret;
assert(__isthreaded);
/* Assign one of the arenas to this thread, in a round-robin fashion. */
malloc_mutex_lock(&arenas_mtx);
ret = arenas[next_arena];
if (ret == NULL)
ret = arenas_extend(next_arena);
if (ret == NULL) {
/*
* Make sure that this function never returns NULL, so that
* choose_arena() doesn't have to check for a NULL return
* value.
*/
ret = arenas[0];
}
next_arena = (next_arena + 1) % narenas;
malloc_mutex_unlock(&arenas_mtx);
arenas_map = ret;
return (ret);
}
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline int
arena_chunk_comp(arena_chunk_t *a, arena_chunk_t *b)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(a != NULL);
assert(b != NULL);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if ((uintptr_t)a < (uintptr_t)b)
return (-1);
else if (a == b)
return (0);
else
return (1);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Generate red-black tree code for arena chunks. */
RB_GENERATE_STATIC(arena_chunk_tree_s, arena_chunk_s, link, arena_chunk_comp);
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
static inline void *
arena_run_reg_alloc(arena_run_t *run, arena_bin_t *bin)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
void *ret;
unsigned i, mask, bit, regind;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(run->magic == ARENA_RUN_MAGIC);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
for (i = run->regs_minelm; i < bin->regs_mask_nelms; i++) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
mask = run->regs_mask[i];
if (mask != 0) {
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/* Usable allocation found. */
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
bit = ffs((int)mask) - 1;
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
regind = ((i << (SIZEOF_INT_2POW + 3)) + bit);
ret = (void *)&((char *)run)[bin->reg0_offset
+ (bin->reg_size * regind)];
/* Clear bit. */
mask ^= (1 << bit);
run->regs_mask[i] = mask;
return (ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
} else {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Make a note that nothing before this element
* contains a free region.
*/
run->regs_minelm = i + 1;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/* Not reached. */
assert(0);
Add an unreachable return statement, in order to avoid a compiler warning for non-standard optimization levels. Reported by: Michael Zach <zach@webges.com>
2006-04-05 18:46:24 +00:00
return (NULL);
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
}
static inline void
arena_run_reg_dalloc(arena_run_t *run, arena_bin_t *bin, void *ptr, size_t size)
{
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
/*
* To divide by a number D that is not a power of two we multiply
* by (2^21 / D) and then right shift by 21 positions.
*
* X / D
*
* becomes
*
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
* (X * size_invs[(D >> QUANTUM_2POW_MIN) - 3]) >> SIZE_INV_SHIFT
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
*/
#define SIZE_INV_SHIFT 21
#define SIZE_INV(s) (((1 << SIZE_INV_SHIFT) / (s << QUANTUM_2POW_MIN)) + 1)
static const unsigned size_invs[] = {
SIZE_INV(3),
SIZE_INV(4), SIZE_INV(5), SIZE_INV(6), SIZE_INV(7),
SIZE_INV(8), SIZE_INV(9), SIZE_INV(10), SIZE_INV(11),
SIZE_INV(12),SIZE_INV(13), SIZE_INV(14), SIZE_INV(15),
SIZE_INV(16),SIZE_INV(17), SIZE_INV(18), SIZE_INV(19),
SIZE_INV(20),SIZE_INV(21), SIZE_INV(22), SIZE_INV(23),
SIZE_INV(24),SIZE_INV(25), SIZE_INV(26), SIZE_INV(27),
SIZE_INV(28),SIZE_INV(29), SIZE_INV(30), SIZE_INV(31)
Conditionally expand the size_invs lookup table in arena_run_reg_dalloc() so that architectures with a quantum of 8 (rather than 16) work. Restore arm's quantum to 8. Submitted by: jmg
2006-07-27 19:09:32 +00:00
#if (QUANTUM_2POW_MIN < 4)
,
SIZE_INV(32), SIZE_INV(33), SIZE_INV(34), SIZE_INV(35),
SIZE_INV(36), SIZE_INV(37), SIZE_INV(38), SIZE_INV(39),
SIZE_INV(40), SIZE_INV(41), SIZE_INV(42), SIZE_INV(43),
SIZE_INV(44), SIZE_INV(45), SIZE_INV(46), SIZE_INV(47),
SIZE_INV(48), SIZE_INV(49), SIZE_INV(50), SIZE_INV(51),
SIZE_INV(52), SIZE_INV(53), SIZE_INV(54), SIZE_INV(55),
SIZE_INV(56), SIZE_INV(57), SIZE_INV(58), SIZE_INV(59),
SIZE_INV(60), SIZE_INV(61), SIZE_INV(62), SIZE_INV(63)
#endif
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
};
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
unsigned diff, regind, elm, bit;
assert(run->magic == ARENA_RUN_MAGIC);
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
assert(((sizeof(size_invs)) / sizeof(unsigned)) + 3
>= (SMALL_MAX_DEFAULT >> QUANTUM_2POW_MIN));
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/*
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
* Avoid doing division with a variable divisor if possible. Using
* actual division here can reduce allocator throughput by over 20%!
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
*/
diff = (unsigned)((uintptr_t)ptr - (uintptr_t)run - bin->reg0_offset);
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
if ((size & (size - 1)) == 0) {
/*
* log2_table allows fast division of a power of two in the
* [1..128] range.
*
* (x / divisor) becomes (x >> log2_table[divisor - 1]).
*/
static const unsigned char log2_table[] = {
0, 1, 0, 2, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7
};
if (size <= 128)
regind = (diff >> log2_table[size - 1]);
else if (size <= 32768)
regind = diff >> (8 + log2_table[(size >> 8) - 1]);
else {
/*
* The page size is too large for us to use the lookup
* table. Use real division.
*/
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
regind = diff / size;
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
}
} else if (size <= ((sizeof(size_invs) / sizeof(unsigned))
<< QUANTUM_2POW_MIN) + 2) {
regind = size_invs[(size >> QUANTUM_2POW_MIN) - 3] * diff;
regind >>= SIZE_INV_SHIFT;
} else {
/*
* size_invs isn't large enough to handle this size class, so
* calculate regind using actual division. This only happens
* if the user increases small_max via the 'S' runtime
* configuration option.
*/
regind = diff / size;
};
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
assert(diff == regind * size);
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
assert(regind < bin->nregs);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
elm = regind >> (SIZEOF_INT_2POW + 3);
if (elm < run->regs_minelm)
run->regs_minelm = elm;
bit = regind - (elm << (SIZEOF_INT_2POW + 3));
assert((run->regs_mask[elm] & (1 << bit)) == 0);
run->regs_mask[elm] |= (1 << bit);
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
#undef SIZE_INV
#undef SIZE_INV_SHIFT
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena_run_split(arena_t *arena, arena_run_t *run, size_t size)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
unsigned run_ind, map_offset, total_pages, need_pages, rem_pages;
unsigned i;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(run);
run_ind = (unsigned)(((uintptr_t)run - (uintptr_t)chunk)
>> pagesize_2pow);
total_pages = chunk->map[run_ind].npages;
need_pages = (size >> pagesize_2pow);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
assert(need_pages <= total_pages);
rem_pages = total_pages - need_pages;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Split enough pages from the front of run to fit allocation size. */
map_offset = run_ind;
for (i = 0; i < need_pages; i++) {
chunk->map[map_offset + i].npages = need_pages;
chunk->map[map_offset + i].pos = i;
}
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Keep track of trailing unused pages for later use. */
if (rem_pages > 0) {
/* Update map for trailing pages. */
map_offset += need_pages;
chunk->map[map_offset].npages = rem_pages;
chunk->map[map_offset].pos = POS_FREE;
chunk->map[map_offset + rem_pages - 1].npages = rem_pages;
chunk->map[map_offset + rem_pages - 1].pos = POS_FREE;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
chunk->pages_used += need_pages;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_chunk_t *
arena_chunk_alloc(arena_t *arena)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
if (arena->spare != NULL) {
chunk = arena->spare;
arena->spare = NULL;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
RB_INSERT(arena_chunk_tree_s, &arena->chunks, chunk);
} else {
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunk = (arena_chunk_t *)chunk_alloc(chunksize);
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
if (chunk == NULL)
return (NULL);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifdef MALLOC_STATS
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena->stats.mapped += chunksize;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
chunk->arena = arena;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
RB_INSERT(arena_chunk_tree_s, &arena->chunks, chunk);
/*
* Claim that no pages are in use, since the header is merely
* overhead.
*/
chunk->pages_used = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
chunk->max_frun_npages = chunk_npages -
arena_chunk_header_npages;
chunk->min_frun_ind = arena_chunk_header_npages;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
/*
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* Initialize enough of the map to support one maximal free run.
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
*/
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
chunk->map[arena_chunk_header_npages].npages = chunk_npages -
arena_chunk_header_npages;
chunk->map[arena_chunk_header_npages].pos = POS_FREE;
chunk->map[chunk_npages - 1].npages = chunk_npages -
arena_chunk_header_npages;
chunk->map[chunk_npages - 1].pos = POS_FREE;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (chunk);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
arena_chunk_dealloc(arena_t *arena, arena_chunk_t *chunk)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
/*
* Remove chunk from the chunk tree, regardless of whether this chunk
* will be cached, so that the arena does not use it.
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
RB_REMOVE(arena_chunk_tree_s, &chunk->arena->chunks, chunk);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
if (opt_hint == false) {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
if (arena->spare != NULL) {
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunk_dealloc((void *)arena->spare, chunksize);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifdef MALLOC_STATS
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena->stats.mapped -= chunksize;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#endif
}
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
arena->spare = chunk;
} else {
assert(arena->spare == NULL);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunk_dealloc((void *)chunk, chunksize);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifdef MALLOC_STATS
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena->stats.mapped -= chunksize;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#endif
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_run_promote(arena_t *arena, arena_bin_t *bin, arena_run_t *run)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(bin == run->bin);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
/* Promote. */
assert(run->free_min > run->nfree);
assert(run->quartile < RUN_Q100);
run->quartile++;
#ifdef MALLOC_STATS
bin->stats.npromote++;
#endif
/* Re-file run. */
switch (run->quartile) {
case RUN_QINIT:
assert(0);
break;
case RUN_Q0:
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_before_insert(arena_bin_link(&bin->runs0), run,
link);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
run->free_max = bin->nregs - 1;
run->free_min = (bin->nregs >> 1) + 1;
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
break;
case RUN_Q25:
qr_remove(run, link);
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_before_insert(arena_bin_link(&bin->runs25), run,
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
link);
run->free_max = ((bin->nregs >> 2) * 3) - 1;
run->free_min = (bin->nregs >> 2) + 1;
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
break;
case RUN_Q50:
qr_remove(run, link);
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_before_insert(arena_bin_link(&bin->runs50), run,
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
link);
run->free_max = (bin->nregs >> 1) - 1;
run->free_min = 1;
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
break;
case RUN_Q75:
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
/*
* Skip RUN_Q75 during promotion from RUN_Q50.
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* Separate handling of RUN_Q75 and RUN_Q100 allows us
* to keep completely full runs in RUN_Q100, thus
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
* guaranteeing that runs in RUN_Q75 are only mostly
* full. This provides a method for avoiding a linear
* search for non-full runs, which avoids some
* pathological edge cases.
*/
run->quartile++;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifdef MALLOC_STATS
/*
* Count as a double promotion, in order to keep
* promotions and demotions symmetric.
*/
bin->stats.npromote++;
#endif
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
/* Fall through. */
case RUN_Q100:
qr_remove(run, link);
assert(bin->runcur == run);
bin->runcur = NULL;
run->free_max = 0;
run->free_min = 0;
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
break;
default:
assert(0);
break;
}
}
static void
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_run_demote(arena_t *arena, arena_bin_t *bin, arena_run_t *run)
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
{
assert(bin == run->bin);
/* Demote. */
assert(run->free_max < run->nfree);
assert(run->quartile > RUN_QINIT);
run->quartile--;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
bin->stats.ndemote++;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Re-file run. */
switch (run->quartile) {
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
case RUN_QINIT:
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
qr_remove(run, link);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
bin->stats.curruns--;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (bin->runcur == run)
bin->runcur = NULL;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_DEBUG
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->magic = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_run_dalloc(arena, run, bin->run_size);
break;
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
case RUN_Q0:
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
qr_remove(run, link);
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_before_insert(arena_bin_link(&bin->runs0), run,
link);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
run->free_max = bin->nregs - 1;
run->free_min = (bin->nregs >> 1) + 1;
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
break;
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
case RUN_Q25:
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
qr_remove(run, link);
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_before_insert(arena_bin_link(&bin->runs25), run,
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
link);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
run->free_max = ((bin->nregs >> 2) * 3) - 1;
run->free_min = (bin->nregs >> 2) + 1;
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
break;
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
case RUN_Q50:
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
qr_remove(run, link);
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_before_insert(arena_bin_link(&bin->runs50), run,
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
link);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
run->free_max = (bin->nregs >> 1) - 1;
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
run->free_min = 1;
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
break;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
case RUN_Q75:
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_before_insert(arena_bin_link(&bin->runs75), run,
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
link);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
run->free_max = (bin->nregs >> 2) - 1;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->free_min = 1;
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
assert(run->nfree <= run->free_max);
assert(run->nfree >= run->free_min);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
break;
case RUN_Q100:
default:
assert(0);
break;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_run_t *
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena_run_alloc(arena_t *arena, size_t size)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
arena_chunk_t *chunk;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena_run_t *run;
unsigned need_npages, limit_pages, compl_need_npages;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert(size <= (chunksize - (arena_chunk_header_npages <<
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
pagesize_2pow)));
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert((size & pagesize_mask) == 0);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* Search through arena's chunks in address order for a free run that is
* large enough. Look for the first fit.
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
*/
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
need_npages = (size >> pagesize_2pow);
limit_pages = chunk_npages - arena_chunk_header_npages;
compl_need_npages = limit_pages - need_npages;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
RB_FOREACH(chunk, arena_chunk_tree_s, &arena->chunks) {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/*
* Avoid searching this chunk if there are not enough
* contiguous free pages for there to possibly be a large
* enough free run.
*/
if (chunk->pages_used <= compl_need_npages &&
need_npages <= chunk->max_frun_npages) {
arena_chunk_map_t *mapelm;
unsigned i;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
unsigned max_frun_npages = 0;
unsigned min_frun_ind = chunk_npages;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
assert(chunk->min_frun_ind >=
arena_chunk_header_npages);
for (i = chunk->min_frun_ind; i < chunk_npages;) {
mapelm = &chunk->map[i];
if (mapelm->pos == POS_FREE) {
if (mapelm->npages >= need_npages) {
run = (arena_run_t *)
((uintptr_t)chunk + (i <<
pagesize_2pow));
/* Update page map. */
arena_run_split(arena, run,
size);
return (run);
}
if (mapelm->npages >
max_frun_npages) {
max_frun_npages =
mapelm->npages;
}
if (i < min_frun_ind) {
min_frun_ind = i;
if (i < chunk->min_frun_ind)
chunk->min_frun_ind = i;
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
i += mapelm->npages;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/*
* Search failure. Reset cached chunk->max_frun_npages.
* chunk->min_frun_ind was already reset above (if
* necessary).
*/
chunk->max_frun_npages = max_frun_npages;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* No usable runs. Allocate a new chunk, then try again. */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
chunk = arena_chunk_alloc(arena);
if (chunk == NULL)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (NULL);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
run = (arena_run_t *)((uintptr_t)chunk + (arena_chunk_header_npages <<
pagesize_2pow));
/* Update page map. */
arena_run_split(arena, run, size);
return (run);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void
arena_run_dalloc(arena_t *arena, arena_run_t *run, size_t size)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
unsigned run_ind, run_pages;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(run);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run_ind = (unsigned)(((uintptr_t)run - (uintptr_t)chunk)
>> pagesize_2pow);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert(run_ind >= arena_chunk_header_npages);
assert(run_ind < (chunksize >> pagesize_2pow));
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run_pages = (size >> pagesize_2pow);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
assert(run_pages == chunk->map[run_ind].npages);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Subtract pages from count of pages used in chunk. */
chunk->pages_used -= run_pages;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Mark run as deallocated. */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
assert(chunk->map[run_ind].npages == run_pages);
chunk->map[run_ind].pos = POS_FREE;
assert(chunk->map[run_ind + run_pages - 1].npages == run_pages);
chunk->map[run_ind + run_pages - 1].pos = POS_FREE;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* Tell the kernel that we don't need the data in this run, but only if
* requested via runtime configuration.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*/
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
if (opt_hint)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
madvise(run, size, MADV_FREE);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Try to coalesce with neighboring runs. */
if (run_ind > arena_chunk_header_npages &&
chunk->map[run_ind - 1].pos == POS_FREE) {
unsigned prev_npages;
/* Coalesce with previous run. */
prev_npages = chunk->map[run_ind - 1].npages;
run_ind -= prev_npages;
assert(chunk->map[run_ind].npages == prev_npages);
assert(chunk->map[run_ind].pos == POS_FREE);
run_pages += prev_npages;
chunk->map[run_ind].npages = run_pages;
assert(chunk->map[run_ind].pos == POS_FREE);
chunk->map[run_ind + run_pages - 1].npages = run_pages;
assert(chunk->map[run_ind + run_pages - 1].pos == POS_FREE);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
if (run_ind + run_pages < chunk_npages &&
chunk->map[run_ind + run_pages].pos == POS_FREE) {
unsigned next_npages;
/* Coalesce with next run. */
next_npages = chunk->map[run_ind + run_pages].npages;
run_pages += next_npages;
assert(chunk->map[run_ind + run_pages - 1].npages ==
next_npages);
assert(chunk->map[run_ind + run_pages - 1].pos == POS_FREE);
chunk->map[run_ind].npages = run_pages;
chunk->map[run_ind].pos = POS_FREE;
chunk->map[run_ind + run_pages - 1].npages = run_pages;
assert(chunk->map[run_ind + run_pages - 1].pos == POS_FREE);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
if (chunk->map[run_ind].npages > chunk->max_frun_npages)
chunk->max_frun_npages = chunk->map[run_ind].npages;
if (run_ind < chunk->min_frun_ind)
chunk->min_frun_ind = run_ind;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Deallocate chunk if it is now completely unused. */
if (chunk->pages_used == 0)
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
arena_chunk_dealloc(arena, chunk);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_run_t *
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_nonfull_run_get(arena_t *arena, arena_bin_t *bin)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
arena_run_t *run;
unsigned i, remainder;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Look for a usable run. */
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
if ((run = qr_next(arena_bin_link(&bin->runs50), link))
!= arena_bin_link(&bin->runs50)
|| (run = qr_next(arena_bin_link(&bin->runs25), link))
!= arena_bin_link(&bin->runs25)
|| (run = qr_next(arena_bin_link(&bin->runs0), link))
!= arena_bin_link(&bin->runs0)
|| (run = qr_next(arena_bin_link(&bin->runs75), link))
!= arena_bin_link(&bin->runs75)) {
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
/* run is guaranteed to have available space. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
qr_remove(run, link);
return (run);
}
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* No existing runs have any space available. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Allocate a new run. */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
run = arena_run_alloc(arena, bin->run_size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (run == NULL)
return (NULL);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Initialize run internals. */
qr_new(run, link);
run->bin = bin;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
for (i = 0; i < bin->regs_mask_nelms; i++)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->regs_mask[i] = UINT_MAX;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
remainder = bin->nregs % (1 << (SIZEOF_INT_2POW + 3));
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (remainder != 0) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/* The last element has spare bits that need to be unset. */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
run->regs_mask[i] = (UINT_MAX >> ((1 << (SIZEOF_INT_2POW + 3))
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
- remainder));
i++;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->regs_minelm = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->nfree = bin->nregs;
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
run->quartile = RUN_QINIT;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->free_max = bin->nregs;
run->free_min = ((bin->nregs >> 2) * 3) + 1;
#ifdef MALLOC_DEBUG
run->magic = ARENA_RUN_MAGIC;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
bin->stats.nruns++;
bin->stats.curruns++;
if (bin->stats.curruns > bin->stats.highruns)
bin->stats.highruns = bin->stats.curruns;
Style changes. Inching closer to convergence with OpenBSD.
2003-10-25 12:56:51 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (run);
}
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* bin->runcur must have space available before this function is called. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline void *
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_malloc_easy(arena_t *arena, arena_bin_t *bin, arena_run_t *run)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
void *ret;
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(run->magic == ARENA_RUN_MAGIC);
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
assert(run->nfree > 0);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
ret = arena_run_reg_alloc(run, bin);
assert(ret != NULL);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->nfree--;
if (run->nfree < run->free_min) {
/* Promote run to higher fullness quartile. */
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_run_promote(arena, bin, run);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
}
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Re-fill bin->runcur, then call arena_bin_malloc_easy(). */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void *
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_malloc_hard(arena_t *arena, arena_bin_t *bin)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
assert(bin->runcur == NULL || bin->runcur->quartile == RUN_Q100);
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
bin->runcur = arena_bin_nonfull_run_get(arena, bin);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (bin->runcur == NULL)
return (NULL);
assert(bin->runcur->magic == ARENA_RUN_MAGIC);
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
assert(bin->runcur->nfree > 0);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
return (arena_bin_malloc_easy(arena, bin, bin->runcur));
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/*
* Calculate bin->run_size such that it meets the following constraints:
*
* *) bin->run_size >= min_run_size
* *) bin->run_size <= arena_maxclass
* *) bin->run_size <= RUN_MAX_SMALL
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* *) run header overhead <= RUN_MAX_OVRHD (or header overhead relaxed).
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
*
* bin->nregs, bin->regs_mask_nelms, and bin->reg0_offset are
* also calculated here, since these settings are all interdependent.
*/
static size_t
arena_bin_run_size_calc(arena_bin_t *bin, size_t min_run_size)
{
size_t try_run_size, good_run_size;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
unsigned good_nregs, good_mask_nelms, good_reg0_offset;
unsigned try_nregs, try_mask_nelms, try_reg0_offset;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
float max_ovrhd = RUN_MAX_OVRHD;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
assert(min_run_size >= pagesize);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
assert(min_run_size <= arena_maxclass);
assert(min_run_size <= RUN_MAX_SMALL);
/*
* Calculate known-valid settings before entering the run_size
* expansion loop, so that the first part of the loop always copies
* valid settings.
*
* The do..while loop iteratively reduces the number of regions until
* the run header and the regions no longer overlap. A closed formula
* would be quite messy, since there is an interdependency between the
* header's mask length and the number of regions.
*/
try_run_size = min_run_size;
try_nregs = ((try_run_size - sizeof(arena_run_t)) / bin->reg_size)
+ 1; /* Counter-act the first line of the loop. */
do {
try_nregs--;
try_mask_nelms = (try_nregs >> (SIZEOF_INT_2POW + 3)) +
((try_nregs & ((1 << (SIZEOF_INT_2POW + 3)) - 1)) ? 1 : 0);
try_reg0_offset = try_run_size - (try_nregs * bin->reg_size);
} while (sizeof(arena_run_t) + (sizeof(unsigned) * (try_mask_nelms - 1))
> try_reg0_offset);
/* run_size expansion loop. */
do {
/*
* Copy valid settings before trying more aggressive settings.
*/
good_run_size = try_run_size;
good_nregs = try_nregs;
good_mask_nelms = try_mask_nelms;
good_reg0_offset = try_reg0_offset;
/* Try more aggressive settings. */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
try_run_size += pagesize;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
try_nregs = ((try_run_size - sizeof(arena_run_t)) /
bin->reg_size) + 1; /* Counter-act try_nregs-- in loop. */
do {
try_nregs--;
try_mask_nelms = (try_nregs >> (SIZEOF_INT_2POW + 3)) +
((try_nregs & ((1 << (SIZEOF_INT_2POW + 3)) - 1)) ?
1 : 0);
try_reg0_offset = try_run_size - (try_nregs *
bin->reg_size);
} while (sizeof(arena_run_t) + (sizeof(unsigned) *
(try_mask_nelms - 1)) > try_reg0_offset);
} while (try_run_size <= arena_maxclass && try_run_size <= RUN_MAX_SMALL
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
&& max_ovrhd > RUN_MAX_OVRHD_RELAX / ((float)(bin->reg_size << 3))
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
&& ((float)(try_reg0_offset)) / ((float)(try_run_size)) >
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
max_ovrhd);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
assert(sizeof(arena_run_t) + (sizeof(unsigned) * (good_mask_nelms - 1))
<= good_reg0_offset);
assert((good_mask_nelms << (SIZEOF_INT_2POW + 3)) >= good_nregs);
/* Copy final settings. */
bin->run_size = good_run_size;
bin->nregs = good_nregs;
bin->regs_mask_nelms = good_mask_nelms;
bin->reg0_offset = good_reg0_offset;
return (good_run_size);
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void *
arena_malloc(arena_t *arena, size_t size)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
void *ret;
Various cosmetics Improve chances of troublefree 64bit operation. [imp] Noticed by: Warner Losh <imp@village.org>
1996-10-26 08:19:07 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(arena != NULL);
assert(arena->magic == ARENA_MAGIC);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(size != 0);
assert(QUANTUM_CEILING(size) <= arena_maxclass);
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (size <= bin_maxclass) {
arena_bin_t *bin;
arena_run_t *run;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Small allocation. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (size < small_min) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Tiny. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
size = pow2_ceil(size);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
bin = &arena->bins[ffs((int)(size >> (TINY_MIN_2POW +
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
1)))];
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
#if (!defined(NDEBUG) || defined(MALLOC_STATS))
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
* Bin calculation is always correct, but we may need
* to fix size for the purposes of assertions and/or
* stats accuracy.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
if (size < (1 << TINY_MIN_2POW))
size = (1 << TINY_MIN_2POW);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#endif
} else if (size <= small_max) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Quantum-spaced. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
size = QUANTUM_CEILING(size);
bin = &arena->bins[ntbins + (size >> opt_quantum_2pow)
- 1];
} else {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Sub-page. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
size = pow2_ceil(size);
bin = &arena->bins[ntbins + nqbins
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
+ (ffs((int)(size >> opt_small_max_2pow)) - 2)];
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
assert(size == bin->reg_size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
malloc_mutex_lock(&arena->mtx);
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
if ((run = bin->runcur) != NULL)
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
ret = arena_bin_malloc_easy(arena, bin, run);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
else
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
ret = arena_bin_malloc_hard(arena, bin);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (ret == NULL) {
malloc_mutex_unlock(&arena->mtx);
return (NULL);
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
bin->stats.nrequests++;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena->stats.nmalloc_small++;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena->stats.allocated_small += size;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#endif
} else {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Large allocation. */
size = PAGE_CEILING(size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
malloc_mutex_lock(&arena->mtx);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
ret = (void *)arena_run_alloc(arena, size);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (ret == NULL) {
malloc_mutex_unlock(&arena->mtx);
return (NULL);
}
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena->stats.nmalloc_large++;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena->stats.allocated_large += size;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
malloc_mutex_unlock(&arena->mtx);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (opt_junk)
memset(ret, 0xa5, size);
else if (opt_zero)
memset(ret, 0, size);
return (ret);
}
static inline void
arena_palloc_trim(arena_t *arena, arena_chunk_t *chunk, unsigned pageind,
unsigned npages)
{
unsigned i;
assert(npages > 0);
/*
* Modifiy the map such that arena_run_dalloc() sees the run as
* separately allocated.
*/
for (i = 0; i < npages; i++) {
chunk->map[pageind + i].npages = npages;
chunk->map[pageind + i].pos = i;
}
arena_run_dalloc(arena, (arena_run_t *)((uintptr_t)chunk + (pageind <<
pagesize_2pow)), npages << pagesize_2pow);
}
/* Only handles large allocations that require more than page alignment. */
static void *
arena_palloc(arena_t *arena, size_t alignment, size_t size, size_t alloc_size)
{
void *ret;
size_t offset;
arena_chunk_t *chunk;
unsigned pageind, i, npages;
assert((size & pagesize_mask) == 0);
assert((alignment & pagesize_mask) == 0);
npages = size >> pagesize_2pow;
malloc_mutex_lock(&arena->mtx);
ret = (void *)arena_run_alloc(arena, alloc_size);
if (ret == NULL) {
malloc_mutex_unlock(&arena->mtx);
return (NULL);
}
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ret);
offset = (uintptr_t)ret & (alignment - 1);
assert((offset & pagesize_mask) == 0);
assert(offset < alloc_size);
if (offset == 0) {
pageind = (((uintptr_t)ret - (uintptr_t)chunk) >>
pagesize_2pow);
/* Update the map for the run to be kept. */
for (i = 0; i < npages; i++) {
chunk->map[pageind + i].npages = npages;
assert(chunk->map[pageind + i].pos == i);
}
/* Trim trailing space. */
arena_palloc_trim(arena, chunk, pageind + npages,
(alloc_size - size) >> pagesize_2pow);
} else {
size_t leadsize, trailsize;
leadsize = alignment - offset;
ret = (void *)((uintptr_t)ret + leadsize);
pageind = (((uintptr_t)ret - (uintptr_t)chunk) >>
pagesize_2pow);
/* Update the map for the run to be kept. */
for (i = 0; i < npages; i++) {
chunk->map[pageind + i].npages = npages;
chunk->map[pageind + i].pos = i;
}
/* Trim leading space. */
arena_palloc_trim(arena, chunk, pageind - (leadsize >>
pagesize_2pow), leadsize >> pagesize_2pow);
trailsize = alloc_size - leadsize - size;
if (trailsize != 0) {
/* Trim trailing space. */
assert(trailsize < alloc_size);
arena_palloc_trim(arena, chunk, pageind + npages,
trailsize >> pagesize_2pow);
}
}
#ifdef MALLOC_STATS
arena->stats.nmalloc_large++;
arena->stats.allocated_large += size;
#endif
malloc_mutex_unlock(&arena->mtx);
if (opt_junk)
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
memset(ret, 0xa5, size);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
else if (opt_zero)
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
memset(ret, 0, size);
return (ret);
}
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Return the size of the allocation pointed to by ptr. */
static size_t
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arena_salloc(const void *ptr)
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
{
size_t ret;
arena_chunk_t *chunk;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena_chunk_map_t *mapelm;
unsigned pageind;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
assert(ptr != NULL);
assert(CHUNK_ADDR2BASE(ptr) != ptr);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
/*
* No arena data structures that we query here can change in a way that
* affects this function, so we don't need to lock.
*/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
pageind = (((uintptr_t)ptr - (uintptr_t)chunk) >> pagesize_2pow);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
mapelm = &chunk->map[pageind];
if (mapelm->pos != 0 || ptr != (void *)((uintptr_t)chunk) + (pageind <<
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
pagesize_2pow)) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
arena_run_t *run;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
pageind -= mapelm->pos;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
run = (arena_run_t *)&((char *)chunk)[pageind << pagesize_2pow];
assert(run->magic == ARENA_RUN_MAGIC);
ret = run->bin->reg_size;
} else
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
ret = mapelm->npages << pagesize_2pow;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
return (ret);
}
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
static void *
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arena_ralloc(void *ptr, size_t size, size_t oldsize)
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
{
void *ret;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Avoid moving the allocation if the size class would not change. */
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
if (size < small_min) {
if (oldsize < small_min &&
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
ffs((int)(pow2_ceil(size) >> (TINY_MIN_2POW + 1)))
== ffs((int)(pow2_ceil(oldsize) >> (TINY_MIN_2POW + 1))))
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
goto IN_PLACE;
} else if (size <= small_max) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
if (oldsize >= small_min && oldsize <= small_max &&
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
(QUANTUM_CEILING(size) >> opt_quantum_2pow)
== (QUANTUM_CEILING(oldsize) >> opt_quantum_2pow))
goto IN_PLACE;
} else {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/*
* We make no attempt to resize runs here, though it would be
* possible to do so.
*/
if (oldsize > small_max && PAGE_CEILING(size) == oldsize)
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
goto IN_PLACE;
}
/*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* If we get here, then size and oldsize are different enough that we
* need to use a different size class. In that case, fall back to
* allocating new space and copying.
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
*/
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = arena_malloc(choose_arena(), size);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
if (ret == NULL)
return (NULL);
if (size < oldsize)
memcpy(ret, ptr, size);
else
memcpy(ret, ptr, oldsize);
idalloc(ptr);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
IN_PLACE:
if (opt_junk && size < oldsize)
memset(&((char *)ptr)[size], 0x5a, oldsize - size);
else if (opt_zero && size > oldsize)
memset(&((char *)ptr)[size], 0, size - oldsize);
return (ptr);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
static void
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arena_dalloc(arena_t *arena, arena_chunk_t *chunk, void *ptr)
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
unsigned pageind;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena_chunk_map_t *mapelm;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
size_t size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(arena != NULL);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(arena->magic == ARENA_MAGIC);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
assert(chunk->arena == arena);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(ptr != NULL);
assert(CHUNK_ADDR2BASE(ptr) != ptr);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
pageind = (((uintptr_t)ptr - (uintptr_t)chunk) >> pagesize_2pow);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
mapelm = &chunk->map[pageind];
if (mapelm->pos != 0 || ptr != (void *)((uintptr_t)chunk) + (pageind <<
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
pagesize_2pow)) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_run_t *run;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arena_bin_t *bin;
Make it possible to override the function which writes messages to stderr in case of warnings and errors. Rename malloc_options to have a leading underscore, I belive I have been told that is more correct namespace wise.
2000-11-26 10:30:18 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Small allocation. */
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
pageind -= mapelm->pos;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run = (arena_run_t *)&((char *)chunk)[pageind << pagesize_2pow];
assert(run->magic == ARENA_RUN_MAGIC);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
bin = run->bin;
size = bin->reg_size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (opt_junk)
memset(ptr, 0x5a, size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
malloc_mutex_lock(&arena->mtx);
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
arena_run_reg_dalloc(run, bin, ptr, size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->nfree++;
if (run->nfree > run->free_max) {
/* Demote run to lower fullness quartile. */
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_run_demote(arena, bin, run);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifdef MALLOC_STATS
arena->stats.allocated_small -= size;
arena->stats.ndalloc_small++;
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
} else {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Large allocation. */
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
size = mapelm->npages << pagesize_2pow;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
assert((((uintptr_t)ptr) & pagesize_mask) == 0);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
if (opt_junk)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
memset(ptr, 0x5a, size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
malloc_mutex_lock(&arena->mtx);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_run_dalloc(arena, (arena_run_t *)ptr, size);
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena->stats.allocated_large -= size;
arena->stats.ndalloc_large++;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#endif
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_unlock(&arena->mtx);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
}
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static bool
arena_new(arena_t *arena)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
unsigned i;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_bin_t *bin;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
size_t pow2_size, prev_run_size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_init(&arena->mtx);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
memset(&arena->stats, 0, sizeof(arena_stats_t));
#endif
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Initialize chunks. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
RB_INIT(&arena->chunks);
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
arena->spare = NULL;
Malloc flag X makes malloc behave like the canonical xmalloc() wrapper. Untested support for Solaris from John-Mark Gurney Reviewed by: phk Submitted by: (partially) John-Mark Gurney <gurney_j@resnet.uoregon.edu>
1997-05-30 20:39:32 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Initialize bins. */
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
prev_run_size = pagesize;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* (2^n)-spaced tiny bins. */
for (i = 0; i < ntbins; i++) {
bin = &arena->bins[i];
bin->runcur = NULL;
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_new(arena_bin_link(&bin->runs0), link);
qr_new(arena_bin_link(&bin->runs25), link);
qr_new(arena_bin_link(&bin->runs50), link);
qr_new(arena_bin_link(&bin->runs75), link);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
bin->reg_size = (1 << (TINY_MIN_2POW + i));
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
prev_run_size = arena_bin_run_size_calc(bin, prev_run_size);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
memset(&bin->stats, 0, sizeof(malloc_bin_stats_t));
#endif
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Quantum-spaced bins. */
for (; i < ntbins + nqbins; i++) {
bin = &arena->bins[i];
bin->runcur = NULL;
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_new(arena_bin_link(&bin->runs0), link);
qr_new(arena_bin_link(&bin->runs25), link);
qr_new(arena_bin_link(&bin->runs50), link);
qr_new(arena_bin_link(&bin->runs75), link);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
bin->reg_size = quantum * (i - ntbins + 1);
pow2_size = pow2_ceil(quantum * (i - ntbins + 1));
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
prev_run_size = arena_bin_run_size_calc(bin, prev_run_size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
memset(&bin->stats, 0, sizeof(malloc_bin_stats_t));
#endif
}
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* (2^n)-spaced sub-page bins. */
for (; i < ntbins + nqbins + nsbins; i++) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
bin = &arena->bins[i];
bin->runcur = NULL;
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
qr_new(arena_bin_link(&bin->runs0), link);
qr_new(arena_bin_link(&bin->runs25), link);
qr_new(arena_bin_link(&bin->runs50), link);
qr_new(arena_bin_link(&bin->runs75), link);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
bin->reg_size = (small_max << (i - (ntbins + nqbins) + 1));
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
prev_run_size = arena_bin_run_size_calc(bin, prev_run_size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
memset(&bin->stats, 0, sizeof(malloc_bin_stats_t));
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_DEBUG
arena->magic = ARENA_MAGIC;
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
return (false);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Create a new arena and insert it into the arenas array at index ind. */
static arena_t *
arenas_extend(unsigned ind)
{
arena_t *ret;
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Allocate enough space for trailing bins. */
ret = (arena_t *)base_alloc(sizeof(arena_t)
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
+ (sizeof(arena_bin_t) * (ntbins + nqbins + nsbins - 1)));
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (ret != NULL && arena_new(ret) == false) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
arenas[ind] = ret;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
return (ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/* Only reached if there is an OOM error. */
Preallocate a small structure, so we can sbrk(2) further back. Reviewed by: phk Submitted by: Kaleb Keithly <kaleb@x.org>
1995-12-18 12:03:54 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/*
* OOM here is quite inconvenient to propagate, since dealing with it
* would require a check for failure in the fast path. Instead, punt
* by using arenas[0]. In practice, this is an extremely unlikely
* failure.
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error initializing arena\n", "", "");
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (opt_abort)
abort();
return (arenas[0]);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* End arena.
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Begin general internal functions.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
static void *
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
huge_malloc(size_t size)
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
size_t csize;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
chunk_node_t *node;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
/* Allocate one or more contiguous chunks for this request. */
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
csize = CHUNK_CEILING(size);
if (csize == 0) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* size is large enough to cause size_t wrap-around. */
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
return (NULL);
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Allocate a chunk node with which to track the chunk. */
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
node = base_chunk_node_alloc();
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
if (node == NULL)
return (NULL);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
ret = chunk_alloc(csize);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (ret == NULL) {
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
base_chunk_node_dealloc(node);
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
return (NULL);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
}
Add a missing case for the switch statement in arena_run_reg_dalloc(). [1] Fix a leak in chunk_dealloc(). [2] Reported by: [1] djam8193ah@hotmail.com, [2] Ville-Pertti Keinonen <will@exomi.com>
2006-06-20 20:38:25 +00:00
/* Insert node into huge. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
node->chunk = ret;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
node->size = csize;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_lock(&chunks_mtx);
RB_INSERT(chunk_tree_s, &huge, node);
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
huge_nmalloc++;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
huge_allocated += csize;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
malloc_mutex_unlock(&chunks_mtx);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (opt_junk)
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
memset(ret, 0xa5, csize);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
else if (opt_zero)
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
memset(ret, 0, csize);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
return (ret);
}
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
/* Only handles large allocations that require more than chunk alignment. */
static void *
huge_palloc(size_t alignment, size_t size)
{
void *ret;
size_t alloc_size, chunk_size, offset;
chunk_node_t *node;
/*
* This allocation requires alignment that is even larger than chunk
* alignment. This means that huge_malloc() isn't good enough.
*
* Allocate almost twice as many chunks as are demanded by the size or
* alignment, in order to assure the alignment can be achieved, then
* unmap leading and trailing chunks.
*/
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
assert(alignment >= chunksize);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunk_size = CHUNK_CEILING(size);
if (size >= alignment)
alloc_size = chunk_size + alignment - chunksize;
else
alloc_size = (alignment << 1) - chunksize;
/* Allocate a chunk node with which to track the chunk. */
node = base_chunk_node_alloc();
if (node == NULL)
return (NULL);
ret = chunk_alloc(alloc_size);
if (ret == NULL) {
base_chunk_node_dealloc(node);
return (NULL);
}
offset = (uintptr_t)ret & (alignment - 1);
assert((offset & chunksize_mask) == 0);
assert(offset < alloc_size);
if (offset == 0) {
/* Trim trailing space. */
chunk_dealloc((void *)((uintptr_t)ret + chunk_size), alloc_size
- chunk_size);
} else {
size_t trailsize;
/* Trim leading space. */
chunk_dealloc(ret, alignment - offset);
ret = (void *)((uintptr_t)ret + (alignment - offset));
trailsize = alloc_size - (alignment - offset) - chunk_size;
if (trailsize != 0) {
/* Trim trailing space. */
assert(trailsize < alloc_size);
chunk_dealloc((void *)((uintptr_t)ret + chunk_size),
trailsize);
}
}
/* Insert node into huge. */
node->chunk = ret;
node->size = chunk_size;
malloc_mutex_lock(&chunks_mtx);
RB_INSERT(chunk_tree_s, &huge, node);
#ifdef MALLOC_STATS
huge_allocated += size;
#endif
malloc_mutex_unlock(&chunks_mtx);
if (opt_junk)
memset(ret, 0xa5, chunk_size);
else if (opt_zero)
memset(ret, 0, chunk_size);
return (ret);
}
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
static void *
huge_ralloc(void *ptr, size_t size, size_t oldsize)
{
void *ret;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Avoid moving the allocation if the size class would not change. */
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
if (oldsize > arena_maxclass &&
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
CHUNK_CEILING(size) == CHUNK_CEILING(oldsize))
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
return (ptr);
/*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* If we get here, then size and oldsize are different enough that we
* need to use a different size class. In that case, fall back to
* allocating new space and copying.
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
*/
ret = huge_malloc(size);
if (ret == NULL)
return (NULL);
if (CHUNK_ADDR2BASE(ptr) == ptr) {
/* The old allocation is a chunk. */
if (size < oldsize)
memcpy(ret, ptr, size);
else
memcpy(ret, ptr, oldsize);
} else {
/* The old allocation is a region. */
assert(oldsize < size);
memcpy(ret, ptr, oldsize);
}
idalloc(ptr);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
}
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void
huge_dalloc(void *ptr)
{
chunk_node_t key;
chunk_node_t *node;
malloc_mutex_lock(&chunks_mtx);
/* Extract from tree of huge allocations. */
key.chunk = ptr;
node = RB_FIND(chunk_tree_s, &huge, &key);
assert(node != NULL);
assert(node->chunk == ptr);
RB_REMOVE(chunk_tree_s, &huge, node);
#ifdef MALLOC_STATS
/* Update counters. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
huge_ndalloc++;
huge_allocated -= node->size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_unlock(&chunks_mtx);
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Unmap chunk. */
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
#ifdef USE_BRK
if (opt_junk)
memset(node->chunk, 0x5a, node->size);
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
chunk_dealloc(node->chunk, node->size);
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
base_chunk_node_dealloc(node);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
static void *
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
imalloc(size_t size)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
void *ret;
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(size != 0);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (size <= arena_maxclass)
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = arena_malloc(choose_arena(), size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
else
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
ret = huge_malloc(size);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
static void *
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ipalloc(size_t alignment, size_t size)
Remove trailing whitespace.
1995-05-30 05:51:47 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
size_t ceil_size;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
* Round size up to the nearest multiple of alignment.
*
* This done, we can take advantage of the fact that for each small
* size class, every object is aligned at the smallest power of two
* that is non-zero in the base two representation of the size. For
* example:
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
*
* Size | Base 2 | Minimum alignment
* -----+----------+------------------
* 96 | 1100000 | 32
* 144 | 10100000 | 32
* 192 | 11000000 | 64
*
* Depending on runtime settings, it is possible that arena_malloc()
* will further round up to a power of two, but that never causes
* correctness issues.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
ceil_size = (size + (alignment - 1)) & (-alignment);
/*
* (ceil_size < size) protects against the combination of maximal
* alignment and size greater than maximal alignment.
*/
if (ceil_size < size) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* size_t overflow. */
return (NULL);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
if (ceil_size <= pagesize || (alignment <= pagesize
&& ceil_size <= arena_maxclass))
ret = arena_malloc(choose_arena(), ceil_size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
else {
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
size_t run_size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
/*
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
* We can't achieve sub-page alignment, so round up alignment
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
* permanently; it makes later calculations simpler.
*/
alignment = PAGE_CEILING(alignment);
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
ceil_size = PAGE_CEILING(size);
/*
* (ceil_size < size) protects against very large sizes within
* pagesize of SIZE_T_MAX.
*
* (ceil_size + alignment < ceil_size) protects against the
* combination of maximal alignment and ceil_size large enough
* to cause overflow. This is similar to the first overflow
* check above, but it needs to be repeated due to the new
* ceil_size value, which may now be *equal* to maximal
* alignment, whereas before we only detected overflow if the
* original size was *greater* than maximal alignment.
*/
if (ceil_size < size || ceil_size + alignment < ceil_size) {
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
/* size_t overflow. */
return (NULL);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
/*
* Calculate the size of the over-size run that arena_palloc()
* would need to allocate in order to guarantee the alignment.
*/
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
if (ceil_size >= alignment)
run_size = ceil_size + alignment - pagesize;
else {
/*
* It is possible that (alignment << 1) will cause
* overflow, but it doesn't matter because we also
* subtract pagesize, which in the case of overflow
* leaves us with a very large run_size. That causes
* the first conditional below to fail, which means
* that the bogus run_size value never gets used for
* anything important.
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
run_size = (alignment << 1) - pagesize;
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
}
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
if (run_size <= arena_maxclass) {
ret = arena_palloc(choose_arena(), alignment, ceil_size,
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
run_size);
} else if (alignment <= chunksize)
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
ret = huge_malloc(ceil_size);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
else
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
ret = huge_palloc(alignment, ceil_size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Use uintptr_t rather than size_t when casting pointers to integers. Also, fix the few remaining casting style(9) errors that remained after the functional change. Reported by: jmallett
2006-01-20 03:11:11 +00:00
assert(((uintptr_t)ret & (alignment - 1)) == 0);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
}
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
static void *
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
icalloc(size_t size)
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (size <= arena_maxclass) {
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = arena_malloc(choose_arena(), size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (ret == NULL)
return (NULL);
memset(ret, 0, size);
} else {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* The virtual memory system provides zero-filled pages, so
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
* there is no need to do so manually, unless opt_junk is
* enabled, in which case huge_malloc() fills huge allocations
* with junk.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
ret = huge_malloc(size);
if (ret == NULL)
return (NULL);
if (opt_junk)
memset(ret, 0, size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef USE_BRK
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
else if ((uintptr_t)ret >= (uintptr_t)brk_base
&& (uintptr_t)ret < (uintptr_t)brk_max) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* This may be a re-used brk chunk. Therefore, zero
* the memory.
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
memset(ret, 0, size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
#endif
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static size_t
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
isalloc(const void *ptr)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
size_t ret;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(ptr != NULL);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
if (chunk != ptr) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Region. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(chunk->arena->magic == ARENA_MAGIC);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = arena_salloc(ptr);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
} else {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
chunk_node_t *node, key;
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Chunk (huge allocation). */
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_lock(&chunks_mtx);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Extract from tree of huge allocations. */
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
key.chunk = __DECONST(void *, ptr);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
node = RB_FIND(chunk_tree_s, &huge, &key);
assert(node != NULL);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
ret = node->size;
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_unlock(&chunks_mtx);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
static void *
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
iralloc(void *ptr, size_t size)
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
{
void *ret;
size_t oldsize;
assert(ptr != NULL);
assert(size != 0);
oldsize = isalloc(ptr);
if (size <= arena_maxclass)
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = arena_ralloc(ptr, size, oldsize);
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
else
ret = huge_ralloc(ptr, size, oldsize);
return (ret);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void
idalloc(void *ptr)
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(ptr != NULL);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
if (chunk != ptr) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Region. */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arena_dalloc(chunk->arena, chunk, ptr);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
} else
huge_dalloc(ptr);
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void
malloc_print_stats(void)
{
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (opt_print_stats) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
char s[UMAX2S_BUFSIZE];
_malloc_message("___ Begin malloc statistics ___\n", "", "",
"");
_malloc_message("Assertions ",
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef NDEBUG
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
"disabled",
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#else
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
"enabled",
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
"\n", "");
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
_malloc_message("Boolean MALLOC_OPTIONS: ",
opt_abort ? "A" : "a",
opt_junk ? "J" : "j",
opt_hint ? "H" : "h");
_malloc_message(opt_utrace ? "PU" : "Pu",
opt_sysv ? "V" : "v",
opt_xmalloc ? "X" : "x",
opt_zero ? "Z\n" : "z\n");
_malloc_message("CPUs: ", umax2s(ncpus, s), "\n", "");
_malloc_message("Max arenas: ", umax2s(narenas, s), "\n", "");
_malloc_message("Pointer size: ", umax2s(sizeof(void *), s),
"\n", "");
_malloc_message("Quantum size: ", umax2s(quantum, s), "\n", "");
_malloc_message("Max small size: ", umax2s(small_max, s), "\n",
"");
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
_malloc_message("Chunk size: ", umax2s(chunksize, s), "", "");
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
_malloc_message(" (2^", umax2s(opt_chunk_2pow, s), ")\n", "");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
{
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
size_t allocated, mapped;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
unsigned i;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
arena_t *arena;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Calculate and print allocated/mapped stats. */
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* arenas. */
for (i = 0, allocated = 0; i < narenas; i++) {
if (arenas[i] != NULL) {
malloc_mutex_lock(&arenas[i]->mtx);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
allocated +=
arenas[i]->stats.allocated_small;
allocated +=
arenas[i]->stats.allocated_large;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
malloc_mutex_unlock(&arenas[i]->mtx);
}
}
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* huge/base. */
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
malloc_mutex_lock(&chunks_mtx);
allocated += huge_allocated;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
mapped = stats_chunks.curchunks * chunksize;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
malloc_mutex_unlock(&chunks_mtx);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_mutex_lock(&base_mtx);
mapped += base_mapped;
malloc_mutex_unlock(&base_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_printf("Allocated: %zu, mapped: %zu\n",
allocated, mapped);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Print chunk stats. */
{
chunk_stats_t chunks_stats;
malloc_mutex_lock(&chunks_mtx);
chunks_stats = stats_chunks;
malloc_mutex_unlock(&chunks_mtx);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_printf("chunks: nchunks "
"highchunks curchunks\n");
malloc_printf(" %13llu%13lu%13lu\n",
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
chunks_stats.nchunks,
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
chunks_stats.highchunks,
chunks_stats.curchunks);
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Print chunk stats. */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_printf(
"huge: nmalloc ndalloc allocated\n");
malloc_printf(" %12llu %12llu %12zu\n",
huge_nmalloc, huge_ndalloc, huge_allocated
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
* chunksize);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Print stats for each arena. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
for (i = 0; i < narenas; i++) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
arena = arenas[i];
if (arena != NULL) {
malloc_printf(
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
"\narenas[%u]:\n", i);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_lock(&arena->mtx);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
stats_print(arena);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_unlock(&arena->mtx);
}
}
}
#endif /* #ifdef MALLOC_STATS */
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message("--- End malloc statistics ---\n", "", "", "");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* FreeBSD's pthreads implementation calls malloc(3), so the malloc
* implementation has to take pains to avoid infinite recursion during
* initialization.
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline bool
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_init(void)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (malloc_initialized == false)
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (malloc_init_hard());
return (false);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
static bool
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_init_hard(void)
{
unsigned i, j;
int linklen;
char buf[PATH_MAX + 1];
const char *opts;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
malloc_mutex_lock(&init_lock);
if (malloc_initialized) {
/*
* Another thread initialized the allocator before this one
* acquired init_lock.
*/
malloc_mutex_unlock(&init_lock);
return (false);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Get number of CPUs. */
{
int mib[2];
size_t len;
mib[0] = CTL_HW;
mib[1] = HW_NCPU;
len = sizeof(ncpus);
if (sysctl(mib, 2, &ncpus, &len, (void *) 0, 0) == -1) {
/* Error. */
ncpus = 1;
}
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Get page size. */
{
long result;
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
result = sysconf(_SC_PAGESIZE);
assert(result != -1);
pagesize = (unsigned) result;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
* We assume that pagesize is a power of 2 when calculating
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* pagesize_mask and pagesize_2pow.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*/
assert(((result - 1) & result) == 0);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
pagesize_mask = result - 1;
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
pagesize_2pow = ffs((int)result) - 1;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
for (i = 0; i < 3; i++) {
/* Get runtime configuration. */
switch (i) {
case 0:
if ((linklen = readlink("/etc/malloc.conf", buf,
sizeof(buf) - 1)) != -1) {
/*
* Use the contents of the "/etc/malloc.conf"
* symbolic link's name.
*/
buf[linklen] = '\0';
opts = buf;
} else {
/* No configuration specified. */
buf[0] = '\0';
opts = buf;
}
break;
case 1:
if (issetugid() == 0 && (opts =
getenv("MALLOC_OPTIONS")) != NULL) {
/*
* Do nothing; opts is already initialized to
* the value of the MALLOC_OPTIONS environment
* variable.
*/
} else {
/* No configuration specified. */
buf[0] = '\0';
opts = buf;
}
break;
case 2:
if (_malloc_options != NULL) {
/*
* Use options that were compiled into the program.
*/
opts = _malloc_options;
} else {
/* No configuration specified. */
buf[0] = '\0';
opts = buf;
}
break;
default:
/* NOTREACHED */
assert(false);
}
for (j = 0; opts[j] != '\0'; j++) {
switch (opts[j]) {
case 'a':
opt_abort = false;
break;
case 'A':
opt_abort = true;
break;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
case 'h':
opt_hint = false;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
break;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
case 'H':
opt_hint = true;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
break;
case 'j':
opt_junk = false;
break;
case 'J':
opt_junk = true;
break;
case 'k':
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
* Run fullness quartile limits don't have
* enough resolution if there are too few
* regions for the largest bin size classes.
*/
Increase the minimum chunk size by a power of two (32kB --> 64kB, assuming 4kB pages), in order to avoid dangerous rounding error when calculating fullness limits during run promotion/demotion. Convert a structure bitfield to a normal field in areana_run_t. This should have been changed along with the other fields in revision 1.120.
2006-05-10 00:07:45 +00:00
if (opt_chunk_2pow > pagesize_2pow + 4)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
opt_chunk_2pow--;
break;
case 'K':
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/*
* There must be fewer pages in a chunk than
* can be recorded by the pos field of
* arena_chunk_map_t, in order to make POS_FREE
* special.
*/
if (opt_chunk_2pow - pagesize_2pow
< (sizeof(uint32_t) << 3) - 1)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
opt_chunk_2pow++;
break;
case 'n':
opt_narenas_lshift--;
break;
case 'N':
opt_narenas_lshift++;
break;
case 'p':
opt_print_stats = false;
break;
case 'P':
opt_print_stats = true;
break;
case 'q':
if (opt_quantum_2pow > QUANTUM_2POW_MIN)
opt_quantum_2pow--;
break;
case 'Q':
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (opt_quantum_2pow < pagesize_2pow - 1)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
opt_quantum_2pow++;
break;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
case 's':
if (opt_small_max_2pow > QUANTUM_2POW_MIN)
opt_small_max_2pow--;
break;
case 'S':
if (opt_small_max_2pow < pagesize_2pow - 1)
opt_small_max_2pow++;
break;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
case 'u':
opt_utrace = false;
break;
case 'U':
opt_utrace = true;
break;
case 'v':
opt_sysv = false;
break;
case 'V':
opt_sysv = true;
break;
case 'x':
opt_xmalloc = false;
break;
case 'X':
opt_xmalloc = true;
break;
case 'z':
opt_zero = false;
break;
case 'Z':
opt_zero = true;
break;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
default: {
char cbuf[2];
cbuf[0] = opts[j];
cbuf[1] = '\0';
_malloc_message(_getprogname(),
": (malloc) Unsupported character in "
"malloc options: '", cbuf, "'\n");
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Take care to call atexit() only once. */
if (opt_print_stats) {
/* Print statistics at exit. */
atexit(malloc_print_stats);
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Set variables according to the value of opt_small_max_2pow. */
if (opt_small_max_2pow < opt_quantum_2pow)
opt_small_max_2pow = opt_quantum_2pow;
small_max = (1 << opt_small_max_2pow);
/* Set bin-related variables. */
bin_maxclass = (pagesize >> 1);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
assert(opt_quantum_2pow >= TINY_MIN_2POW);
ntbins = opt_quantum_2pow - TINY_MIN_2POW;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(ntbins <= opt_quantum_2pow);
nqbins = (small_max >> opt_quantum_2pow);
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
nsbins = pagesize_2pow - opt_small_max_2pow - 1;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Set variables according to the value of opt_quantum_2pow. */
quantum = (1 << opt_quantum_2pow);
quantum_mask = quantum - 1;
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
if (ntbins > 0)
small_min = (quantum >> 1) + 1;
else
small_min = 1;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(small_min <= quantum);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Set variables according to the value of opt_chunk_2pow. */
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunksize = (1LU << opt_chunk_2pow);
chunksize_mask = chunksize - 1;
chunk_npages = (chunksize >> pagesize_2pow);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
{
unsigned header_size;
header_size = sizeof(arena_chunk_t) + (sizeof(arena_chunk_map_t)
* (chunk_npages - 1));
arena_chunk_header_npages = (header_size >> pagesize_2pow);
if ((header_size & pagesize_mask) != 0)
arena_chunk_header_npages++;
}
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena_maxclass = chunksize - (arena_chunk_header_npages <<
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
pagesize_2pow);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(0, 0, 0);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
memset(&stats_chunks, 0, sizeof(chunk_stats_t));
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Various sanity checks that regard configuration. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(quantum >= sizeof(void *));
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(quantum <= pagesize);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert(chunksize >= pagesize);
assert(quantum * 4 <= chunksize);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Initialize chunks data. */
malloc_mutex_init(&chunks_mtx);
RB_INIT(&huge);
#ifdef USE_BRK
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
malloc_mutex_init(&brk_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
brk_base = sbrk(0);
brk_prev = brk_base;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
brk_max = brk_base;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
huge_nmalloc = 0;
huge_ndalloc = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
huge_allocated = 0;
#endif
RB_INIT(&old_chunks);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/* Initialize base allocation data structures. */
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
base_mapped = 0;
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
#endif
#ifdef USE_BRK
/*
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
* Allocate a base chunk here, since it doesn't actually have to be
* chunk-aligned. Doing this before allocating any other chunks allows
* the use of space that would otherwise be wasted.
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
*/
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
base_chunk_alloc(0);
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
#endif
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
base_chunk_nodes = NULL;
malloc_mutex_init(&base_mtx);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (ncpus > 1) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* For SMP systems, create four times as many arenas as there
* are CPUs by default.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
opt_narenas_lshift += 2;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Determine how many arenas to use. */
Fix calculation of the number of arenas to use on multi-processor systems.
2006-02-04 01:11:30 +00:00
narenas = ncpus;
Allow the 'n' option to decrease the number of arenas below the default, to as little as one arena. Also, limit the number of arenas to avoid a potential invariant violation in base_alloc().
2006-03-26 23:41:35 +00:00
if (opt_narenas_lshift > 0) {
if ((narenas << opt_narenas_lshift) > narenas)
narenas <<= opt_narenas_lshift;
/*
* Make sure not to exceed the limits of what base_malloc()
* can handle.
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (narenas * sizeof(arena_t *) > chunksize)
narenas = chunksize / sizeof(arena_t *);
Allow the 'n' option to decrease the number of arenas below the default, to as little as one arena. Also, limit the number of arenas to avoid a potential invariant violation in base_alloc().
2006-03-26 23:41:35 +00:00
} else if (opt_narenas_lshift < 0) {
if ((narenas << opt_narenas_lshift) < narenas)
narenas <<= opt_narenas_lshift;
/* Make sure there is at least one arena. */
if (narenas == 0)
narenas = 1;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef NO_TLS
if (narenas > 1) {
static const unsigned primes[] = {1, 3, 5, 7, 11, 13, 17, 19,
23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83,
89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149,
151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211,
223, 227, 229, 233, 239, 241, 251, 257, 263};
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
unsigned nprimes, parenas;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Pick a prime number of hash arenas that is more than narenas
* so that direct hashing of pthread_self() pointers tends to
* spread allocations evenly among the arenas.
*/
assert((narenas & 1) == 0); /* narenas must be even. */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
nprimes = (sizeof(primes) >> SIZEOF_INT_2POW);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
parenas = primes[nprimes - 1]; /* In case not enough primes. */
for (i = 1; i < nprimes; i++) {
if (primes[i] > narenas) {
parenas = primes[i];
break;
}
}
narenas = parenas;
}
#endif
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifndef NO_TLS
next_arena = 0;
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Allocate and initialize arenas. */
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
arenas = (arena_t **)base_alloc(sizeof(arena_t *) * narenas);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
if (arenas == NULL) {
malloc_mutex_unlock(&init_lock);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (true);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Zero the array. In practice, this should always be pre-zeroed,
* since it was just mmap()ed, but let's be sure.
*/
memset(arenas, 0, sizeof(arena_t *) * narenas);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Initialize one arena here. The rest are lazily created in
* arena_choose_hard().
*/
arenas_extend(0);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
if (arenas[0] == NULL) {
malloc_mutex_unlock(&init_lock);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (true);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
}
Isolated all the crap for thread-safe so I can see what goes on again...
1996-07-03 05:03:07 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_init(&arenas_mtx);
Rearrange (centralize) initialization of mallocs internals to always be done before the first call, even if this is a malloc(0) call. PR: 62859
2004-03-07 20:41:27 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_initialized = true;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
malloc_mutex_unlock(&init_lock);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (false);
Isolated all the crap for thread-safe so I can see what goes on again...
1996-07-03 05:03:07 +00:00
}
Integrate calloc with the rest of the gang. Various portability and stylistic cleanups. Kill MALLOC_STATS & the 'D' option. Fix the 'V' option. Major overhaul of the man-page. You milage should not vary. Reviewed by: Keith Bostic <bostic@bostic.com> Submitted by: Keith Bostic <bostic@bostic.com>
1997-06-22 17:54:27 +00:00
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* End general internal functions.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
/******************************************************************************/
/*
* Begin malloc(3)-compatible functions.
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
*/
void *
malloc(size_t size)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
if (malloc_init()) {
ret = NULL;
goto RETURN;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (size == 0) {
if (opt_sysv == false)
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
size = 1;
else {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
ret = NULL;
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
goto RETURN;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = imalloc(size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
RETURN:
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in malloc(): out of memory\n", "",
"");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
abort();
}
errno = ENOMEM;
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(0, size, ret);
return (ret);
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
int
posix_memalign(void **memptr, size_t alignment, size_t size)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
int ret;
void *result;
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
if (malloc_init())
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
result = NULL;
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
else {
/* Make sure that alignment is a large enough power of 2. */
if (((alignment - 1) & alignment) != 0
|| alignment < sizeof(void *)) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in posix_memalign(): "
"invalid alignment\n", "", "");
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
abort();
}
result = NULL;
ret = EINVAL;
goto RETURN;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
result = ipalloc(alignment, size);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (result == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in posix_memalign(): out of memory\n",
"", "");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
abort();
}
ret = ENOMEM;
goto RETURN;
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*memptr = result;
ret = 0;
RETURN:
UTRACE(0, size, result);
return (ret);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
void *
calloc(size_t num, size_t size)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
size_t num_size;
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
if (malloc_init()) {
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
num_size = 0;
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
ret = NULL;
goto RETURN;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
num_size = num * size;
if (num_size == 0) {
Correctly handle the case in calloc(num, size) where (size_t)(num * size) == 0 but both num and size are nonzero. Reported by: Ilja van Sprundel Approved by: jasone Security: Integer overflow; calloc was allocating 1 byte in response to a request for a multiple of 2^32 (or 2^64) bytes instead of returning NULL.
2006-08-13 21:54:47 +00:00
if ((opt_sysv == false) && ((num == 0) || (size == 0)))
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
num_size = 1;
else {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
ret = NULL;
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
goto RETURN;
}
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/*
* Try to avoid division here. We know that it isn't possible to
* overflow during multiplication if neither operand uses any of the
* most significant half of the bits in a size_t.
*/
} else if (((num | size) & (SIZE_T_MAX << (sizeof(size_t) << 2)))
&& (num_size / size != num)) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* size_t overflow. */
ret = NULL;
goto RETURN;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = icalloc(num_size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
RETURN:
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in calloc(): out of memory\n", "",
"");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
abort();
}
errno = ENOMEM;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
UTRACE(0, num_size, ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *
realloc(void *ptr, size_t size)
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (size == 0) {
if (opt_sysv == false)
size = 1;
else {
if (ptr != NULL)
idalloc(ptr);
ret = NULL;
goto RETURN;
}
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ptr != NULL) {
assert(malloc_initialized);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
ret = iralloc(ptr, size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in realloc(): out of "
"memory\n", "", "");
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
abort();
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
}
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
errno = ENOMEM;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
} else {
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (malloc_init())
ret = NULL;
else
ret = imalloc(size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in realloc(): out of "
"memory\n", "", "");
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
abort();
}
errno = ENOMEM;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
RETURN:
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(ptr, size, ret);
return (ret);
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void
free(void *ptr)
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(ptr, 0, 0);
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ptr != NULL) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(malloc_initialized);
idalloc(ptr);
}
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* End malloc(3)-compatible functions.
*/
/******************************************************************************/
Add malloc_usable_size(3). Discussed with: arch@
2006-03-28 22:16:04 +00:00
/*
* Begin non-standard functions.
*/
size_t
malloc_usable_size(const void *ptr)
{
assert(ptr != NULL);
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
return (isalloc(ptr));
Add malloc_usable_size(3). Discussed with: arch@
2006-03-28 22:16:04 +00:00
}
/*
* End non-standard functions.
*/
/******************************************************************************/
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
/*
* Begin library-private functions, used by threading libraries for protection
* of malloc during fork(). These functions are only called if the program is
* running in threaded mode, so there is no need to check whether the program
* is threaded here.
*/
void
_malloc_prefork(void)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
unsigned i;
/* Acquire all mutexes in a safe order. */
malloc_mutex_lock(&arenas_mtx);
for (i = 0; i < narenas; i++) {
if (arenas[i] != NULL)
malloc_mutex_lock(&arenas[i]->mtx);
}
malloc_mutex_unlock(&arenas_mtx);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_lock(&base_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_lock(&chunks_mtx);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
void
_malloc_postfork(void)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
unsigned i;
/* Release all mutexes, now that fork() has completed. */
malloc_mutex_unlock(&chunks_mtx);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_unlock(&base_mtx);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_mutex_lock(&arenas_mtx);
for (i = 0; i < narenas; i++) {
if (arenas[i] != NULL)
malloc_mutex_unlock(&arenas[i]->mtx);
}
malloc_mutex_unlock(&arenas_mtx);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* End library-private functions.
*/
/******************************************************************************/
Reference in New Issue Copy Permalink