d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8530137252
freebsd-nq/lib/libc/stdlib/malloc.c

5591 lines
138 KiB
C
Raw Normal View History

Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*-
Enable both sbrk(2)- and mmap(2)-based memory acquisition methods by default. This has the disadvantage of rendering the datasize resource limit irrelevant, but without this change, legitimate uses of more memory than will fit in the data segment are thwarted by default. Fix chunk_alloc_mmap() to work correctly if initial mapping is not chunk-aligned and mapping extension fails.
2008-01-03 23:22:13 +00:00
* Copyright (C) 2006-2008 Jason Evans <jasone@FreeBSD.org>.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice(s), this list of conditions and the following disclaimer as
* the first lines of this file unmodified other than the possible
* addition of one or more copyright notices.
* 2. Redistributions in binary form must reproduce the above copyright
* notice(s), this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
* BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
* OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
* EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*******************************************************************************
*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* This allocator implementation is designed to provide scalable performance
* for multi-threaded programs on multi-processor systems. The following
* features are included for this purpose:
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
* + Multiple arenas are used if there are multiple CPUs, which reduces lock
* contention and cache sloshing.
*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* + Thread-specific caching is used if there are multiple threads, which
* reduces the amount of locking.
*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* + Cache line sharing between arenas is avoided for internal data
* structures.
*
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
* + Memory is managed in chunks and runs (chunks can be split into runs),
* rather than as individual pages. This provides a constant-time
* mechanism for associating allocations with particular arenas.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* Allocation requests are rounded up to the nearest size class, and no record
* of the original request size is maintained. Allocations are broken into
* categories according to size class. Assuming runtime defaults, 4 kB pages
Remove stale #include <machine/atomic.h>, which as needed by lazy deallocation.
2008-03-07 16:54:03 +00:00
* and a 16 byte quantum on a 32-bit system, the size classes in each category
* are as follows:
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* |=======================================|
* | Category | Subcategory | Size |
* |=======================================|
* | Small | Tiny | 2 |
* | | | 4 |
* | | | 8 |
* | |------------------+---------|
* | | Quantum-spaced | 16 |
* | | | 32 |
* | | | 48 |
* | | | ... |
* | | | 96 |
* | | | 112 |
* | | | 128 |
* | |------------------+---------|
* | | Cacheline-spaced | 192 |
* | | | 256 |
* | | | 320 |
* | | | 384 |
* | | | 448 |
* | | | 512 |
* | |------------------+---------|
* | | Sub-page | 760 |
* | | | 1024 |
* | | | 1280 |
* | | | ... |
* | | | 3328 |
* | | | 3584 |
* | | | 3840 |
* |=======================================|
* | Large | 4 kB |
* | | 8 kB |
* | | 12 kB |
* | | ... |
* | | 1012 kB |
* | | 1016 kB |
* | | 1020 kB |
* |=======================================|
* | Huge | 1 MB |
* | | 2 MB |
* | | 3 MB |
* | | ... |
* |=======================================|
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
*
* A different mechanism is used for each category:
*
* Small : Each size class is segregated into its own set of runs. Each run
* maintains a bitmap of which regions are free/allocated.
*
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
* Large : Each allocation is backed by a dedicated run. Metadata are stored
* in the associated arena chunk header maps.
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
*
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
* Huge : Each allocation is backed by a dedicated contiguous set of chunks.
* Metadata are stored in a separate red-black tree.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
*******************************************************************************
*/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* MALLOC_PRODUCTION disables assertions and statistics gathering. It also
* defaults the A and J runtime options to off. These settings are appropriate
* for production systems.
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
*/
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
/* #define MALLOC_PRODUCTION */
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifndef MALLOC_PRODUCTION
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
/*
* MALLOC_DEBUG enables assertions and other sanity checks, and disables
* inline functions.
*/
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
# define MALLOC_DEBUG
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
/* MALLOC_STATS enables statistics calculation. */
# define MALLOC_STATS
Add NO_MALLOC_EXTRAS, so that various extra features that can cause performance degradation can be disabled via something like the following in /etc/malloc.conf: CFLAGS+=-DNO_MALLOC_EXTRAS Suggested by: deischen
2006-01-27 04:42:10 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/*
* MALLOC_TINY enables support for tiny objects, which are smaller than one
* quantum.
*/
#define MALLOC_TINY
/*
* MALLOC_MAG enables a magazine-based thread-specific caching layer for small
* objects. This makes it possible to allocate/deallocate objects without any
* locking when the cache is in the steady state.
*/
#define MALLOC_MAG
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
/*
* MALLOC_BALANCE enables monitoring of arena lock contention and dynamically
* re-balances arena load if exponentially averaged contention exceeds a
* certain threshold.
*/
#define MALLOC_BALANCE
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
/*
* MALLOC_DSS enables use of sbrk(2) to allocate chunks from the data storage
* segment (DSS). In an ideal world, this functionality would be completely
* unnecessary, but we are burdened by history and the lack of resource limits
* for anonymous mapped memory.
*/
Enable both sbrk(2)- and mmap(2)-based memory acquisition methods by default. This has the disadvantage of rendering the datasize resource limit irrelevant, but without this change, legitimate uses of more memory than will fit in the data segment are thwarted by default. Fix chunk_alloc_mmap() to work correctly if initial mapping is not chunk-aligned and mapping extension fails.
2008-01-03 23:22:13 +00:00
#define MALLOC_DSS
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
Fix the style of the SCM ID's. I believe have made all of libc .c's as consistent as possible.
2002-03-22 21:53:29 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#include "libc_private.h"
#ifdef MALLOC_DEBUG
# define _LOCK_DEBUG
#endif
#include "spinlock.h"
#include "namespace.h"
#include <sys/mman.h>
#include <sys/param.h>
#include <sys/stddef.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/sysctl.h>
#include <sys/uio.h>
#include <sys/ktrace.h> /* Must come after several other sys/ includes. */
#include <machine/cpufunc.h>
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
#include <machine/param.h>
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#include <machine/vmparam.h>
#include <errno.h>
#include <limits.h>
#include <pthread.h>
#include <sched.h>
#include <stdarg.h>
#include <stdbool.h>
#include <stdio.h>
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#include <stdint.h>
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <unistd.h>
#include "un-namespace.h"
Use size_t instead of unsigned for pagesize-related values, in order to avoid downcasting issues. In particular, this change fixes posix_memalign(3) for alignments greater than 2^31 on LP64 systems. Make sure that NDEBUG is always set to be compatible with MALLOC_DEBUG. [1] Reported by: [1] Lee Hyo geol <hyogeollee@gmail.com>
2007-03-29 21:07:17 +00:00
#ifdef MALLOC_DEBUG
# ifdef NDEBUG
# undef NDEBUG
# endif
#else
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# ifndef NDEBUG
# define NDEBUG
# endif
#endif
#include <assert.h>
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
#include "rb.h"
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_DEBUG
/* Disable inlining to make debugging easier. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
# define inline
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
/* Size of stack-allocated buffer passed to strerror_r(). */
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define STRERROR_BUF 64
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/*
* Minimum alignment of allocations is 2^QUANTUM_2POW bytes.
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef __i386__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 2
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
# define CPU_SPINWAIT __asm__ volatile("pause")
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef __ia64__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef __alpha__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define NO_TLS
#endif
#ifdef __sparc64__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define NO_TLS
#endif
#ifdef __amd64__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 3
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
# define CPU_SPINWAIT __asm__ volatile("pause")
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef __arm__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 3
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 2
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define NO_TLS
#endif
Set QUANTUM_2POW_MIN and SIZEOF_PTR_2POW parameters for MIPS Approved by: imp
2008-04-29 22:56:05 +00:00
#ifdef __mips__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 3
Set QUANTUM_2POW_MIN and SIZEOF_PTR_2POW parameters for MIPS Approved by: imp
2008-04-29 22:56:05 +00:00
# define SIZEOF_PTR_2POW 2
# define NO_TLS
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef __powerpc__
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
# define QUANTUM_2POW 4
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
# define SIZEOF_PTR_2POW 2
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#define QUANTUM ((size_t)(1U << QUANTUM_2POW))
#define QUANTUM_MASK (QUANTUM - 1)
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
#define SIZEOF_PTR (1U << SIZEOF_PTR_2POW)
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
/* sizeof(int) == (1U << SIZEOF_INT_2POW). */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#ifndef SIZEOF_INT_2POW
# define SIZEOF_INT_2POW 2
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* We can't use TLS in non-PIC programs, since TLS relies on loader magic. */
#if (!defined(PIC) && !defined(NO_TLS))
# define NO_TLS
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef NO_TLS
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/* MALLOC_MAG requires TLS. */
# ifdef MALLOC_MAG
# undef MALLOC_MAG
# endif
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
/* MALLOC_BALANCE requires TLS. */
# ifdef MALLOC_BALANCE
# undef MALLOC_BALANCE
# endif
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Size and alignment of memory chunks that are allocated by the OS's virtual
* memory system.
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define CHUNK_2POW_DEFAULT 20
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Maximum number of dirty pages per arena. */
#define DIRTY_MAX_DEFAULT (1U << 9)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* Maximum size of L1 cache line. This is used to avoid cache line aliasing.
* In addition, this controls the spacing of cacheline-spaced size classes.
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#define CACHELINE_2POW 6
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
#define CACHELINE ((size_t)(1U << CACHELINE_2POW))
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#define CACHELINE_MASK (CACHELINE - 1)
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/*
* Subpages are an artificially designated partitioning of pages. Their only
* purpose is to support subpage-spaced size classes.
*
* There must be at least 4 subpages per page, due to the way size classes are
* handled.
*/
#define SUBPAGE_2POW 8
#define SUBPAGE ((size_t)(1U << SUBPAGE_2POW))
#define SUBPAGE_MASK (SUBPAGE - 1)
#ifdef MALLOC_TINY
/* Smallest size class to support. */
# define TINY_MIN_2POW 1
#endif
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
* Maximum size class that is a multiple of the quantum, but not (necessarily)
* a power of 2. Above this size, allocations are rounded up to the nearest
* power of 2.
*/
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#define QSPACE_MAX_2POW_DEFAULT 7
/*
* Maximum size class that is a multiple of the cacheline, but not (necessarily)
* a power of 2. Above this size, allocations are rounded up to the nearest
* power of 2.
*/
#define CSPACE_MAX_2POW_DEFAULT 9
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/*
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
* RUN_MAX_OVRHD indicates maximum desired run header overhead. Runs are sized
* as small as possible such that this setting is still honored, without
* violating other constraints. The goal is to make runs as small as possible
* without exceeding a per run external fragmentation threshold.
*
* We use binary fixed point math for overhead computations, where the binary
* point is implicitly RUN_BFP bits to the left.
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
* Note that it is possible to set RUN_MAX_OVRHD low enough that it cannot be
* honored for some/all object sizes, since there is one bit of header overhead
* per object (plus a constant). This constraint is relaxed (ignored) for runs
* that are so small that the per-region overhead is greater than:
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
*
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
* (RUN_MAX_OVRHD / (reg_size << (3+RUN_BFP))
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
*/
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
#define RUN_BFP 12
/* \/ Implicit binary fixed point. */
#define RUN_MAX_OVRHD 0x0000003dU
#define RUN_MAX_OVRHD_RELAX 0x00001800U
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Put a cap on small object run size. This overrides RUN_MAX_OVRHD. */
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
#define RUN_MAX_SMALL (12 * PAGE_SIZE)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
/*
* Hyper-threaded CPUs may need a special instruction inside spin loops in
* order to yield to another virtual CPU. If no such instruction is defined
* above, make CPU_SPINWAIT a no-op.
*/
#ifndef CPU_SPINWAIT
# define CPU_SPINWAIT
#endif
/*
* Adaptive spinning must eventually switch to blocking, in order to avoid the
* potential for priority inversion deadlock. Backing off past a certain point
* can actually waste time.
*/
#define SPIN_LIMIT_2POW 11
/*
* Conversion from spinning to blocking is expensive; we use (1U <<
* BLOCK_COST_2POW) to estimate how many more times costly blocking is than
* worst-case spinning.
*/
#define BLOCK_COST_2POW 4
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
/*
* Default magazine size, in bytes. max_rounds is calculated to make
* optimal use of the space, leaving just enough room for the magazine
* header.
*/
# define MAG_SIZE_2POW_DEFAULT 9
#endif
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
/*
* We use an exponential moving average to track recent lock contention,
* where the size of the history window is N, and alpha=2/(N+1).
*
* Due to integer math rounding, very small values here can cause
* substantial degradation in accuracy, thus making the moving average decay
* faster than it would with precise calculation.
*/
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
# define BALANCE_ALPHA_INV_2POW 9
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
/*
* Threshold value for the exponential moving contention average at which to
* re-assign a thread.
*/
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
# define BALANCE_THRESHOLD_DEFAULT (1U << (SPIN_LIMIT_2POW-4))
#endif
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
Malloc flag X makes malloc behave like the canonical xmalloc() wrapper. Untested support for Solaris from John-Mark Gurney Reviewed by: phk Submitted by: (partially) John-Mark Gurney <gurney_j@resnet.uoregon.edu>
1997-05-30 20:39:32 +00:00
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
/*
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
* Mutexes based on spinlocks. We can't use normal pthread spinlocks in all
* places, because they require malloc()ed memory, which causes bootstrapping
* issues in some cases.
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
typedef struct {
spinlock_t lock;
} malloc_mutex_t;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
/* Set to true once the allocator has been initialized. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static bool malloc_initialized = false;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
/* Used to avoid initialization races. */
static malloc_mutex_t init_lock = {_SPINLOCK_INITIALIZER};
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
/*
* Statistics data structures.
*/
#ifdef MALLOC_STATS
typedef struct malloc_bin_stats_s malloc_bin_stats_t;
struct malloc_bin_stats_s {
/*
* Number of allocation requests that corresponded to the size of this
* bin.
*/
uint64_t nrequests;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
/* Number of magazine reloads from this bin. */
uint64_t nmags;
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Total number of runs created for this bin's size class. */
uint64_t nruns;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
* Total number of runs reused by extracting them from the runs tree for
* this bin's size class.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
uint64_t reruns;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* High-water mark for this bin. */
unsigned long highruns;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Current number of runs in this bin. */
unsigned long curruns;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
typedef struct arena_stats_s arena_stats_t;
struct arena_stats_s {
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Number of bytes currently mapped. */
size_t mapped;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/*
* Total number of purge sweeps, total number of madvise calls made,
* and total pages purged in order to keep dirty unused memory under
* control.
*/
uint64_t npurge;
uint64_t nmadvise;
uint64_t purged;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Per-size-category statistics. */
size_t allocated_small;
uint64_t nmalloc_small;
uint64_t ndalloc_small;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
size_t allocated_large;
uint64_t nmalloc_large;
uint64_t ndalloc_large;
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
/* Number of times this arena reassigned a thread due to contention. */
uint64_t nbalance;
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
typedef struct chunk_stats_s chunk_stats_t;
struct chunk_stats_s {
/* Number of chunks that were allocated. */
uint64_t nchunks;
/* High-water mark for number of chunks allocated. */
unsigned long highchunks;
/*
* Current number of chunks allocated. This value isn't maintained for
* any other purpose, so keep track of it in order to be able to set
* highchunks.
*/
unsigned long curchunks;
};
#endif /* #ifdef MALLOC_STATS */
/******************************************************************************/
/*
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
* Extent data structures.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Tree of extents. */
typedef struct extent_node_s extent_node_t;
struct extent_node_s {
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
#ifdef MALLOC_DSS
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Linkage for the size/address-ordered tree. */
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
rb_node(extent_node_t) link_szad;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
#endif
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Linkage for the address-ordered tree. */
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
rb_node(extent_node_t) link_ad;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Pointer to the extent that this tree node is responsible for. */
void *addr;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Total region size. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
size_t size;
};
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
typedef rb_tree(extent_node_t) extent_tree_t;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Arena data structures.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
typedef struct arena_s arena_t;
typedef struct arena_bin_s arena_bin_t;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Each element of the chunk map corresponds to one page within the chunk. */
typedef struct arena_chunk_map_s arena_chunk_map_t;
struct arena_chunk_map_s {
/*
* Linkage for run trees. There are two disjoint uses:
*
* 1) arena_t's runs_avail tree.
* 2) arena_run_t conceptually uses this linkage for in-use non-full
* runs, rather than directly embedding linkage.
*/
rb_node(arena_chunk_map_t) link;
/*
* Run address (or size) and various flags are stored together. The bit
* layout looks like (assuming 32-bit system):
*
* ???????? ???????? ????---- ---kdzla
*
* ? : Unallocated: Run address for first/last pages, unset for internal
* pages.
* Small: Run address.
* Large: Run size for first page, unset for trailing pages.
* - : Unused.
* k : key?
* d : dirty?
* z : zeroed?
* l : large?
* a : allocated?
*
* Following are example bit patterns for the three types of runs.
*
* r : run address
* s : run size
* x : don't care
* - : 0
* [dzla] : bit set
*
* Unallocated:
* ssssssss ssssssss ssss---- --------
* xxxxxxxx xxxxxxxx xxxx---- ----d---
* ssssssss ssssssss ssss---- -----z--
*
* Small:
* rrrrrrrr rrrrrrrr rrrr---- -------a
* rrrrrrrr rrrrrrrr rrrr---- -------a
* rrrrrrrr rrrrrrrr rrrr---- -------a
*
* Large:
* ssssssss ssssssss ssss---- ------la
* -------- -------- -------- ------la
* -------- -------- -------- ------la
*/
size_t bits;
#define CHUNK_MAP_KEY ((size_t)0x10U)
#define CHUNK_MAP_DIRTY ((size_t)0x08U)
#define CHUNK_MAP_ZEROED ((size_t)0x04U)
#define CHUNK_MAP_LARGE ((size_t)0x02U)
#define CHUNK_MAP_ALLOCATED ((size_t)0x01U)
};
typedef rb_tree(arena_chunk_map_t) arena_avail_tree_t;
typedef rb_tree(arena_chunk_map_t) arena_run_tree_t;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Arena chunk header. */
typedef struct arena_chunk_s arena_chunk_t;
struct arena_chunk_s {
/* Arena that owns the chunk. */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_t *arena;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
/* Linkage for the arena's chunks_dirty tree. */
rb_node(arena_chunk_t) link_dirty;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Number of dirty pages. */
size_t ndirty;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Map of pages within chunk that keeps track of free/large/small. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_map_t map[1]; /* Dynamically sized. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
typedef rb_tree(arena_chunk_t) arena_chunk_tree_t;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
typedef struct arena_run_s arena_run_t;
struct arena_run_s {
#ifdef MALLOC_DEBUG
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
uint32_t magic;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
# define ARENA_RUN_MAGIC 0x384adf93
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Bin this run is associated with. */
arena_bin_t *bin;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Index of first element that might have a free region. */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
unsigned regs_minelm;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Number of free regions in run. */
Increase the minimum chunk size by a power of two (32kB --> 64kB, assuming 4kB pages), in order to avoid dangerous rounding error when calculating fullness limits during run promotion/demotion. Convert a structure bitfield to a normal field in areana_run_t. This should have been changed along with the other fields in revision 1.120.
2006-05-10 00:07:45 +00:00
unsigned nfree;
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/* Bitmask of in-use regions (0: in use, 1: free). */
unsigned regs_mask[1]; /* Dynamically sized. */
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
};
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
struct arena_bin_s {
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Current run being used to service allocations of this bin's size
* class.
*/
arena_run_t *runcur;
/*
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
* Tree of non-full runs. This tree is used when looking for an
* existing run when runcur is no longer usable. We choose the
* non-full run that is lowest in memory; this policy tends to keep
* objects packed well, and it can also help reduce the number of
* almost-empty chunks.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
arena_run_tree_t runs;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Size of regions in a run for this bin's size class. */
size_t reg_size;
/* Total size of a run for this bin's size class. */
size_t run_size;
/* Total number of regions in a run for this bin's size class. */
uint32_t nregs;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/* Number of elements in a run's regs_mask for this bin's size class. */
uint32_t regs_mask_nelms;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Offset of first region in a run for this bin's size class. */
uint32_t reg0_offset;
#ifdef MALLOC_STATS
/* Bin statistics. */
malloc_bin_stats_t stats;
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
};
struct arena_s {
#ifdef MALLOC_DEBUG
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
uint32_t magic;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
# define ARENA_MAGIC 0x947d3d24
#endif
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
/* All operations on this arena require that lock be locked. */
pthread_mutex_t lock;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
arena_stats_t stats;
#endif
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Tree of dirty-page-containing chunks this arena manages. */
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
arena_chunk_tree_t chunks_dirty;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
/*
* In order to avoid rapid chunk allocation/deallocation when an arena
* oscillates right on the cusp of needing a new chunk, cache the most
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
* recently freed chunk. The spare is left in the arena's chunk trees
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
* until it is deleted.
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
*
* There is one spare chunk per arena, rather than one spare total, in
* order to avoid interactions between multiple threads that could make
* a single spare inadequate.
*/
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
arena_chunk_t *spare;
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/*
* Current count of pages within unused runs that are potentially
* dirty, and for which madvise(... MADV_FREE) has not been called. By
* tracking this, we can institute a limit on how much dirty unused
* memory is mapped for each arena.
*/
size_t ndirty;
/*
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
* Size/address-ordered tree of this arena's available runs. This tree
* is used for first-best-fit run allocation.
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_avail_tree_t runs_avail;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
/*
* The arena load balancing machinery needs to keep track of how much
* lock contention there is. This value is exponentially averaged.
*/
uint32_t contention;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#endif
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* bins is used to store rings of free regions of the following sizes,
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
* assuming a 16-byte quantum, 4kB page size, and default
* MALLOC_OPTIONS.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*
* bins[i] | size |
* --------+------+
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* 0 | 2 |
* 1 | 4 |
* 2 | 8 |
* --------+------+
* 3 | 16 |
* 4 | 32 |
* 5 | 48 |
* 6 | 64 |
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* : :
* : :
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* 33 | 496 |
* 34 | 512 |
* --------+------+
* 35 | 1024 |
* 36 | 2048 |
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* --------+------+
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_bin_t bins[1]; /* Dynamically sized. */
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
};
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/******************************************************************************/
/*
* Magazine data structures.
*/
#ifdef MALLOC_MAG
typedef struct mag_s mag_t;
struct mag_s {
size_t binind; /* Index of associated bin. */
size_t nrounds;
void *rounds[1]; /* Dynamically sized. */
};
/*
* Magazines are lazily allocated, but once created, they remain until the
* associated mag_rack is destroyed.
*/
typedef struct bin_mags_s bin_mags_t;
struct bin_mags_s {
mag_t *curmag;
mag_t *sparemag;
};
typedef struct mag_rack_s mag_rack_t;
struct mag_rack_s {
bin_mags_t bin_mags[1]; /* Dynamically sized. */
};
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
/*
* Data.
*/
/* Number of CPUs. */
static unsigned ncpus;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Various bin-related settings. */
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_TINY /* Number of (2^n)-spaced tiny bins. */
# define ntbins ((unsigned)(QUANTUM_2POW - TINY_MIN_2POW))
#else
# define ntbins 0
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static unsigned nqbins; /* Number of quantum-spaced bins. */
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
static unsigned ncbins; /* Number of cacheline-spaced bins. */
static unsigned nsbins; /* Number of subpage-spaced bins. */
static unsigned nbins;
#ifdef MALLOC_TINY
# define tspace_max ((size_t)(QUANTUM >> 1))
#endif
#define qspace_min QUANTUM
static size_t qspace_max;
static size_t cspace_min;
static size_t cspace_max;
static size_t sspace_min;
static size_t sspace_max;
#define bin_maxclass sspace_max
static uint8_t const *size2bin;
/*
* const_size2bin is a static constant lookup table that in the common case can
* be used as-is for size2bin. For dynamically linked programs, this avoids
* a page of memory overhead per process.
*/
#define S2B_1(i) i,
#define S2B_2(i) S2B_1(i) S2B_1(i)
#define S2B_4(i) S2B_2(i) S2B_2(i)
#define S2B_8(i) S2B_4(i) S2B_4(i)
#define S2B_16(i) S2B_8(i) S2B_8(i)
#define S2B_32(i) S2B_16(i) S2B_16(i)
#define S2B_64(i) S2B_32(i) S2B_32(i)
#define S2B_128(i) S2B_64(i) S2B_64(i)
#define S2B_256(i) S2B_128(i) S2B_128(i)
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
static const uint8_t const_size2bin[PAGE_SIZE - 255] = {
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
S2B_1(0xffU) /* 0 */
#if (QUANTUM_2POW == 4)
/* 64-bit system ************************/
# ifdef MALLOC_TINY
S2B_2(0) /* 2 */
S2B_2(1) /* 4 */
S2B_4(2) /* 8 */
S2B_8(3) /* 16 */
# define S2B_QMIN 3
# else
S2B_16(0) /* 16 */
# define S2B_QMIN 0
# endif
S2B_16(S2B_QMIN + 1) /* 32 */
S2B_16(S2B_QMIN + 2) /* 48 */
S2B_16(S2B_QMIN + 3) /* 64 */
S2B_16(S2B_QMIN + 4) /* 80 */
S2B_16(S2B_QMIN + 5) /* 96 */
S2B_16(S2B_QMIN + 6) /* 112 */
S2B_16(S2B_QMIN + 7) /* 128 */
# define S2B_CMIN (S2B_QMIN + 8)
#else
/* 32-bit system ************************/
# ifdef MALLOC_TINY
S2B_2(0) /* 2 */
S2B_2(1) /* 4 */
S2B_4(2) /* 8 */
# define S2B_QMIN 2
# else
S2B_8(0) /* 8 */
# define S2B_QMIN 0
# endif
S2B_8(S2B_QMIN + 1) /* 16 */
S2B_8(S2B_QMIN + 2) /* 24 */
S2B_8(S2B_QMIN + 3) /* 32 */
S2B_8(S2B_QMIN + 4) /* 40 */
S2B_8(S2B_QMIN + 5) /* 48 */
S2B_8(S2B_QMIN + 6) /* 56 */
S2B_8(S2B_QMIN + 7) /* 64 */
S2B_8(S2B_QMIN + 8) /* 72 */
S2B_8(S2B_QMIN + 9) /* 80 */
S2B_8(S2B_QMIN + 10) /* 88 */
S2B_8(S2B_QMIN + 11) /* 96 */
S2B_8(S2B_QMIN + 12) /* 104 */
S2B_8(S2B_QMIN + 13) /* 112 */
S2B_8(S2B_QMIN + 14) /* 120 */
S2B_8(S2B_QMIN + 15) /* 128 */
# define S2B_CMIN (S2B_QMIN + 16)
#endif
/****************************************/
S2B_64(S2B_CMIN + 0) /* 192 */
S2B_64(S2B_CMIN + 1) /* 256 */
S2B_64(S2B_CMIN + 2) /* 320 */
S2B_64(S2B_CMIN + 3) /* 384 */
S2B_64(S2B_CMIN + 4) /* 448 */
S2B_64(S2B_CMIN + 5) /* 512 */
# define S2B_SMIN (S2B_CMIN + 6)
S2B_256(S2B_SMIN + 0) /* 768 */
S2B_256(S2B_SMIN + 1) /* 1024 */
S2B_256(S2B_SMIN + 2) /* 1280 */
S2B_256(S2B_SMIN + 3) /* 1536 */
S2B_256(S2B_SMIN + 4) /* 1792 */
S2B_256(S2B_SMIN + 5) /* 2048 */
S2B_256(S2B_SMIN + 6) /* 2304 */
S2B_256(S2B_SMIN + 7) /* 2560 */
S2B_256(S2B_SMIN + 8) /* 2816 */
S2B_256(S2B_SMIN + 9) /* 3072 */
S2B_256(S2B_SMIN + 10) /* 3328 */
S2B_256(S2B_SMIN + 11) /* 3584 */
S2B_256(S2B_SMIN + 12) /* 3840 */
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
#if (PAGE_SHIFT == 13)
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
S2B_256(S2B_SMIN + 13) /* 4096 */
S2B_256(S2B_SMIN + 14) /* 4352 */
S2B_256(S2B_SMIN + 15) /* 4608 */
S2B_256(S2B_SMIN + 16) /* 4864 */
S2B_256(S2B_SMIN + 17) /* 5120 */
S2B_256(S2B_SMIN + 18) /* 5376 */
S2B_256(S2B_SMIN + 19) /* 5632 */
S2B_256(S2B_SMIN + 20) /* 5888 */
S2B_256(S2B_SMIN + 21) /* 6144 */
S2B_256(S2B_SMIN + 22) /* 6400 */
S2B_256(S2B_SMIN + 23) /* 6656 */
S2B_256(S2B_SMIN + 24) /* 6912 */
S2B_256(S2B_SMIN + 25) /* 7168 */
S2B_256(S2B_SMIN + 26) /* 7424 */
S2B_256(S2B_SMIN + 27) /* 7680 */
S2B_256(S2B_SMIN + 28) /* 7936 */
#endif
};
#undef S2B_1
#undef S2B_2
#undef S2B_4
#undef S2B_8
#undef S2B_16
#undef S2B_32
#undef S2B_64
#undef S2B_128
#undef S2B_256
#undef S2B_QMIN
#undef S2B_CMIN
#undef S2B_SMIN
#ifdef MALLOC_MAG
static size_t max_rounds;
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Various chunk-related settings. */
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
static size_t chunksize;
static size_t chunksize_mask; /* (chunksize - 1). */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static size_t chunk_npages;
static size_t arena_chunk_header_npages;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static size_t arena_maxclass; /* Max size class for arenas. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/********/
/*
* Chunks.
*/
/* Protects chunk-related data structures. */
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
static malloc_mutex_t huge_mtx;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Tree of chunks that are stand-alone huge allocations. */
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
static extent_tree_t huge;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
/*
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
* Protects sbrk() calls. This avoids malloc races among threads, though it
* does not protect against races with threads that call sbrk() directly.
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
*/
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
static malloc_mutex_t dss_mtx;
/* Base address of the DSS. */
static void *dss_base;
/* Current end of the DSS, or ((void *)-1) if the DSS is exhausted. */
static void *dss_prev;
/* Current upper limit on DSS addresses. */
static void *dss_max;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
/*
* Trees of chunks that were previously allocated (trees differ only in node
* ordering). These are used when allocating chunks, in an attempt to re-use
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
* address space. Depending on function, different tree orderings are needed,
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
* which is why there are two trees with the same contents.
*/
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
static extent_tree_t dss_chunks_szad;
static extent_tree_t dss_chunks_ad;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef MALLOC_STATS
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
/* Huge allocation statistics. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static uint64_t huge_nmalloc;
static uint64_t huge_ndalloc;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static size_t huge_allocated;
#endif
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/****************************/
/*
* base (internal allocation).
*/
/*
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
* Current pages that are being used for internal memory allocations. These
* pages are carved up in cacheline-size quanta, so that there is no chance of
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* false cache line sharing.
*/
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
static void *base_pages;
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
static void *base_next_addr;
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
static void *base_past_addr; /* Addr immediately past base_pages. */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static extent_node_t *base_nodes;
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
static malloc_mutex_t base_mtx;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
static size_t base_mapped;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/********/
/*
* Arenas.
*/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Arenas that are used to service external requests. Not all elements of the
* arenas array are necessarily used; arenas are created lazily as needed.
*/
static arena_t **arenas;
static unsigned narenas;
#ifndef NO_TLS
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
# ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
static unsigned narenas_2pow;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
# else
static unsigned next_arena;
# endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static pthread_mutex_t arenas_lock; /* Protects arenas initialization. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifndef NO_TLS
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Map of pthread_self() --> arenas[???], used for selecting an arena to use
* for allocations.
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
static __thread arena_t *arenas_map;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
/*
* Map of thread-specific magazine racks, used for thread-specific object
* caching.
*/
static __thread mag_rack_t *mag_rack;
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
/* Chunk statistics. */
Make all internal variables and functions static. Reported by: ache
2006-01-19 07:23:13 +00:00
static chunk_stats_t stats_chunks;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
/*******************************/
/*
* Runtime configuration options.
*/
const char *_malloc_options;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifndef MALLOC_PRODUCTION
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static bool opt_abort = true;
static bool opt_junk = true;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#else
static bool opt_abort = false;
static bool opt_junk = false;
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
static bool opt_dss = true;
Enable both sbrk(2)- and mmap(2)-based memory acquisition methods by default. This has the disadvantage of rendering the datasize resource limit irrelevant, but without this change, legitimate uses of more memory than will fit in the data segment are thwarted by default. Fix chunk_alloc_mmap() to work correctly if initial mapping is not chunk-aligned and mapping extension fails.
2008-01-03 23:22:13 +00:00
static bool opt_mmap = true;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#endif
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
static bool opt_mag = true;
static size_t opt_mag_size_2pow = MAG_SIZE_2POW_DEFAULT;
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static size_t opt_dirty_max = DIRTY_MAX_DEFAULT;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
static uint64_t opt_balance_threshold = BALANCE_THRESHOLD_DEFAULT;
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static bool opt_print_stats = false;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
static size_t opt_qspace_max_2pow = QSPACE_MAX_2POW_DEFAULT;
static size_t opt_cspace_max_2pow = CSPACE_MAX_2POW_DEFAULT;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static size_t opt_chunk_2pow = CHUNK_2POW_DEFAULT;
static bool opt_utrace = false;
static bool opt_sysv = false;
static bool opt_xmalloc = false;
static bool opt_zero = false;
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
static int opt_narenas_lshift = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
typedef struct {
void *p;
size_t s;
void *r;
} malloc_utrace_t;
#define UTRACE(a, b, c) \
if (opt_utrace) { \
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_utrace_t ut; \
ut.p = (a); \
ut.s = (b); \
ut.r = (c); \
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
utrace(&ut, sizeof(ut)); \
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* Begin function prototypes for non-inline static functions.
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
*/
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
static void malloc_mutex_init(malloc_mutex_t *mutex);
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
static bool malloc_spin_init(pthread_mutex_t *lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void wrtmessage(const char *p1, const char *p2, const char *p3,
const char *p4);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#ifdef MALLOC_STATS
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void malloc_printf(const char *format, ...);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#endif
static char *umax2s(uintmax_t x, char *s);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#ifdef MALLOC_DSS
static bool base_pages_alloc_dss(size_t minsize);
#endif
static bool base_pages_alloc_mmap(size_t minsize);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
static bool base_pages_alloc(size_t minsize);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static void *base_alloc(size_t size);
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
static void *base_calloc(size_t number, size_t size);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static extent_node_t *base_node_alloc(void);
static void base_node_dealloc(extent_node_t *node);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void stats_print(arena_t *arena);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
static void *pages_map(void *addr, size_t size);
static void pages_unmap(void *addr, size_t size);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#ifdef MALLOC_DSS
static void *chunk_alloc_dss(size_t size);
static void *chunk_recycle_dss(size_t size, bool zero);
#endif
static void *chunk_alloc_mmap(size_t size);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
static void *chunk_alloc(size_t size, bool zero);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#ifdef MALLOC_DSS
static extent_node_t *chunk_dealloc_dss_record(void *chunk, size_t size);
static bool chunk_dealloc_dss(void *chunk, size_t size);
#endif
static void chunk_dealloc_mmap(void *chunk, size_t size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void chunk_dealloc(void *chunk, size_t size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#ifndef NO_TLS
static arena_t *choose_arena_hard(void);
#endif
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
static void arena_run_split(arena_t *arena, arena_run_t *run, size_t size,
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
bool large, bool zero);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_chunk_t *arena_chunk_alloc(arena_t *arena);
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
static void arena_chunk_dealloc(arena_t *arena, arena_chunk_t *chunk);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
static arena_run_t *arena_run_alloc(arena_t *arena, size_t size, bool large,
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
bool zero);
static void arena_purge(arena_t *arena);
static void arena_run_dalloc(arena_t *arena, arena_run_t *run, bool dirty);
static void arena_run_trim_head(arena_t *arena, arena_chunk_t *chunk,
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_t *run, size_t oldsize, size_t newsize);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void arena_run_trim_tail(arena_t *arena, arena_chunk_t *chunk,
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_t *run, size_t oldsize, size_t newsize, bool dirty);
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
static arena_run_t *arena_bin_nonfull_run_get(arena_t *arena, arena_bin_t *bin);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
static void *arena_bin_malloc_hard(arena_t *arena, arena_bin_t *bin);
static size_t arena_bin_run_size_calc(arena_bin_t *bin, size_t min_run_size);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#ifdef MALLOC_BALANCE
static void arena_lock_balance_hard(arena_t *arena);
#endif
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
static void mag_load(mag_t *mag);
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void *arena_malloc_large(arena_t *arena, size_t size, bool zero);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
static void *arena_palloc(arena_t *arena, size_t alignment, size_t size,
size_t alloc_size);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
static size_t arena_salloc(const void *ptr);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
static void mag_unload(mag_t *mag);
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void arena_dalloc_large(arena_t *arena, arena_chunk_t *chunk,
void *ptr);
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
static void arena_ralloc_large_shrink(arena_t *arena, arena_chunk_t *chunk,
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
void *ptr, size_t size, size_t oldsize);
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
static bool arena_ralloc_large_grow(arena_t *arena, arena_chunk_t *chunk,
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
void *ptr, size_t size, size_t oldsize);
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
static bool arena_ralloc_large(void *ptr, size_t size, size_t oldsize);
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
static void *arena_ralloc(void *ptr, size_t size, size_t oldsize);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static bool arena_new(arena_t *arena);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static arena_t *arenas_extend(unsigned ind);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
static mag_t *mag_create(arena_t *arena, size_t binind);
static void mag_destroy(mag_t *mag);
static mag_rack_t *mag_rack_create(arena_t *arena);
static void mag_rack_destroy(mag_rack_t *rack);
#endif
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
static void *huge_malloc(size_t size, bool zero);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
static void *huge_palloc(size_t alignment, size_t size);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
static void *huge_ralloc(void *ptr, size_t size, size_t oldsize);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void huge_dalloc(void *ptr);
static void malloc_print_stats(void);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_DEBUG
static void size2bin_validate(void);
#endif
static bool size2bin_init(void);
static bool size2bin_init_hard(void);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
static bool malloc_init_hard(void);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* End function prototypes.
*/
/******************************************************************************/
/*
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
* Begin mutex. We can't use normal pthread mutexes in all places, because
* they require malloc()ed memory, which causes bootstrapping issues in some
* cases.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
static void
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
malloc_mutex_init(malloc_mutex_t *mutex)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
static const spinlock_t lock = _SPINLOCK_INITIALIZER;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
mutex->lock = lock;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline void
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
malloc_mutex_lock(malloc_mutex_t *mutex)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (__isthreaded)
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
_SPINLOCK(&mutex->lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline void
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
malloc_mutex_unlock(malloc_mutex_t *mutex)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (__isthreaded)
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
_SPINUNLOCK(&mutex->lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
/*
* End mutex.
*/
/******************************************************************************/
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
/*
* Begin spin lock. Spin locks here are actually adaptive mutexes that block
* after a period of spinning, because unbounded spinning would allow for
* priority inversion.
*/
/*
* We use an unpublished interface to initialize pthread mutexes with an
* allocation callback, in order to avoid infinite recursion.
*/
int _pthread_mutex_init_calloc_cb(pthread_mutex_t *mutex,
void *(calloc_cb)(size_t, size_t));
__weak_reference(_pthread_mutex_init_calloc_cb_stub,
_pthread_mutex_init_calloc_cb);
int
_pthread_mutex_init_calloc_cb_stub(pthread_mutex_t *mutex,
void *(calloc_cb)(size_t, size_t))
{
return (0);
}
static bool
malloc_spin_init(pthread_mutex_t *lock)
{
if (_pthread_mutex_init_calloc_cb(lock, base_calloc) != 0)
return (true);
return (false);
}
static inline unsigned
malloc_spin_lock(pthread_mutex_t *lock)
{
unsigned ret = 0;
if (__isthreaded) {
if (_pthread_mutex_trylock(lock) != 0) {
Do not spin when trying to lock on a single-CPU system. Reported by: davidxu
2008-11-30 05:55:24 +00:00
/* Exponentially back off if there are multiple CPUs. */
if (ncpus > 1) {
unsigned i;
volatile unsigned j;
for (i = 1; i <= SPIN_LIMIT_2POW; i++) {
for (j = 0; j < (1U << i); j++) {
ret++;
CPU_SPINWAIT;
}
if (_pthread_mutex_trylock(lock) == 0)
return (ret);
Move CPU_SPINWAIT into the innermost spin loop, in order to allow faster preemption while busy-waiting. Submitted by: Mike Schuster <schuster@adobe.com>
2008-08-14 17:31:42 +00:00
}
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
}
/*
* Spinning failed. Block until the lock becomes
* available, in order to avoid indefinite priority
* inversion.
*/
_pthread_mutex_lock(lock);
Adjust an assertion to handle the case where a lock is contested, but spinning is avoided due to running on a single-CPU system. Reported by: stefanf
2008-11-30 19:30:31 +00:00
assert((ret << BLOCK_COST_2POW) != 0 || ncpus == 1);
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
return (ret << BLOCK_COST_2POW);
}
}
return (ret);
}
static inline void
malloc_spin_unlock(pthread_mutex_t *lock)
{
if (__isthreaded)
_pthread_mutex_unlock(lock);
}
/*
* End spin lock.
*/
/******************************************************************************/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Begin Utility functions/macros.
*/
/* Return the chunk address for allocation address a. */
#define CHUNK_ADDR2BASE(a) \
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
((void *)((uintptr_t)(a) & ~chunksize_mask))
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Return the chunk offset of address a. */
#define CHUNK_ADDR2OFFSET(a) \
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
((size_t)((uintptr_t)(a) & chunksize_mask))
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Return the smallest chunk multiple that is >= s. */
#define CHUNK_CEILING(s) \
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
(((s) + chunksize_mask) & ~chunksize_mask)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/* Return the smallest quantum multiple that is >= a. */
#define QUANTUM_CEILING(a) \
(((a) + QUANTUM_MASK) & ~QUANTUM_MASK)
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/* Return the smallest cacheline multiple that is >= s. */
#define CACHELINE_CEILING(s) \
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
(((s) + CACHELINE_MASK) & ~CACHELINE_MASK)
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/* Return the smallest subpage multiple that is >= s. */
#define SUBPAGE_CEILING(s) \
(((s) + SUBPAGE_MASK) & ~SUBPAGE_MASK)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
/* Return the smallest PAGE_SIZE multiple that is >= s. */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#define PAGE_CEILING(s) \
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(((s) + PAGE_MASK) & ~PAGE_MASK)
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_TINY
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Compute the smallest power of 2 that is >= x. */
static inline size_t
pow2_ceil(size_t x)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
x--;
x |= x >> 1;
x |= x >> 2;
x |= x >> 4;
x |= x >> 8;
x |= x >> 16;
#if (SIZEOF_PTR == 8)
x |= x >> 32;
#endif
x++;
return (x);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Remove support for lazy deallocation. Benchmarks across a wide range of allocation patterns, number of CPUs, and MALLOC_OPTIONS settings indicate that lazy deallocation has the potential to worsen throughput dramatically. Performance degradation occurs when multiple threads try to clear the lazy free cache simultaneously. Various experiments to avoid this bottleneck failed to completely solve this problem, while adding yet more complexity.
2008-02-17 17:09:24 +00:00
#ifdef MALLOC_BALANCE
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
/*
* Use a simple linear congruential pseudo-random number generator:
*
* prn(y) = (a*x + c) % m
*
* where the following constants ensure maximal period:
*
* a == Odd number (relatively prime to 2^n), and (a-1) is a multiple of 4.
* c == Odd number (relatively prime to 2^n).
* m == 2^32
*
* See Knuth's TAOCP 3rd Ed., Vol. 2, pg. 17 for details on these constraints.
*
* This choice of m has the disadvantage that the quality of the bits is
* proportional to bit position. For example. the lowest bit has a cycle of 2,
* the next has a cycle of 4, etc. For this reason, we prefer to use the upper
* bits.
*/
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
# define PRN_DEFINE(suffix, var, a, c) \
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
static inline void \
sprn_##suffix(uint32_t seed) \
{ \
var = seed; \
} \
\
static inline uint32_t \
prn_##suffix(uint32_t lg_range) \
{ \
uint32_t ret, x; \
\
assert(lg_range > 0); \
assert(lg_range <= 32); \
\
x = (var * (a)) + (c); \
var = x; \
ret = x >> (32 - lg_range); \
\
return (ret); \
}
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
# define SPRN(suffix, seed) sprn_##suffix(seed)
# define PRN(suffix, lg_range) prn_##suffix(lg_range)
#endif
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
/* Define the PRNG used for arena assignment. */
static __thread uint32_t balance_x;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
PRN_DEFINE(balance, balance_x, 1297, 1301)
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void
wrtmessage(const char *p1, const char *p2, const char *p3, const char *p4)
{
_write(STDERR_FILENO, p1, strlen(p1));
_write(STDERR_FILENO, p2, strlen(p2));
_write(STDERR_FILENO, p3, strlen(p3));
_write(STDERR_FILENO, p4, strlen(p4));
}
void (*_malloc_message)(const char *p1, const char *p2, const char *p3,
const char *p4) = wrtmessage;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#ifdef MALLOC_STATS
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Print to stderr in such a way as to (hopefully) avoid memory allocation.
*/
static void
malloc_printf(const char *format, ...)
{
char buf[4096];
va_list ap;
va_start(ap, format);
vsnprintf(buf, sizeof(buf), format, ap);
va_end(ap);
_malloc_message(buf, "", "", "");
}
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
#endif
/*
* We don't want to depend on vsnprintf() for production builds, since that can
* cause unnecessary bloat for static binaries. umax2s() provides minimal
* integer printing functionality, so that malloc_printf() use can be limited to
* MALLOC_STATS code.
*/
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
#define UMAX2S_BUFSIZE 21
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
static char *
umax2s(uintmax_t x, char *s)
{
unsigned i;
/* Make sure UMAX2S_BUFSIZE is large enough. */
assert(sizeof(uintmax_t) <= 8);
i = UMAX2S_BUFSIZE - 1;
s[i] = '\0';
do {
i--;
s[i] = "0123456789"[x % 10];
x /= 10;
} while (x > 0);
return (&s[i]);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/******************************************************************************/
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static bool
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
base_pages_alloc_dss(size_t minsize)
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
{
/*
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
* Do special DSS allocation here, since base allocations don't need to
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
* be chunk-aligned.
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
*/
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_lock(&dss_mtx);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
if (dss_prev != (void *)-1) {
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
intptr_t incr;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
size_t csize = CHUNK_CEILING(minsize);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
do {
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
/* Get the current end of the DSS. */
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
dss_max = sbrk(0);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
/*
* Calculate how much padding is necessary to
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
* chunk-align the end of the DSS. Don't worry about
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
* dss_max not being chunk-aligned though.
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
incr = (intptr_t)chunksize
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
- (intptr_t)CHUNK_ADDR2OFFSET(dss_max);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(incr >= 0);
if ((size_t)incr < minsize)
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
incr += csize;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
dss_prev = sbrk(incr);
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
if (dss_prev == dss_max) {
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
/* Success. */
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
dss_max = (void *)((intptr_t)dss_prev + incr);
base_pages = dss_prev;
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
base_next_addr = base_pages;
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
base_past_addr = dss_max;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
base_mapped += incr;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#endif
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
return (false);
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
} while (dss_prev != (void *)-1);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
}
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
return (true);
}
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static bool
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
base_pages_alloc_mmap(size_t minsize)
{
size_t csize;
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
assert(minsize != 0);
csize = PAGE_CEILING(minsize);
base_pages = pages_map(NULL, csize);
if (base_pages == NULL)
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
return (true);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
base_next_addr = base_pages;
base_past_addr = (void *)((uintptr_t)base_pages + csize);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#ifdef MALLOC_STATS
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
base_mapped += csize;
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
return (false);
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
static bool
base_pages_alloc(size_t minsize)
{
#ifdef MALLOC_DSS
if (opt_mmap && minsize != 0)
#endif
{
if (base_pages_alloc_mmap(minsize) == false)
return (false);
}
Revert to preferring mmap(2) over sbrk(2) when mapping memory, due to potential extreme contention in the kernel for multi-threaded applications on SMP systems. Reported by: kris
2008-11-03 21:17:18 +00:00
#ifdef MALLOC_DSS
if (opt_dss) {
if (base_pages_alloc_dss(minsize) == false)
return (false);
}
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
return (true);
}
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static void *
base_alloc(size_t size)
{
void *ret;
size_t csize;
/* Round size up to nearest multiple of the cacheline size. */
csize = CACHELINE_CEILING(size);
malloc_mutex_lock(&base_mtx);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
/* Make sure there's enough space for the allocation. */
if ((uintptr_t)base_next_addr + csize > (uintptr_t)base_past_addr) {
In the error path through base_alloc(), release base_mtx [1]. Fix bit vector initialization for run headers. Submitted by: [1] Mike Schuster <schuster@adobe.com>
2008-06-10 15:46:18 +00:00
if (base_pages_alloc(csize)) {
malloc_mutex_unlock(&base_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
return (NULL);
In the error path through base_alloc(), release base_mtx [1]. Fix bit vector initialization for run headers. Submitted by: [1] Mike Schuster <schuster@adobe.com>
2008-06-10 15:46:18 +00:00
}
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
}
/* Allocate. */
ret = base_next_addr;
Use uintptr_t rather than size_t when casting pointers to integers. Also, fix the few remaining casting style(9) errors that remained after the functional change. Reported by: jmallett
2006-01-20 03:11:11 +00:00
base_next_addr = (void *)((uintptr_t)base_next_addr + csize);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_unlock(&base_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
return (ret);
}
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
static void *
base_calloc(size_t number, size_t size)
{
void *ret;
ret = base_alloc(number * size);
memset(ret, 0, number * size);
return (ret);
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static extent_node_t *
base_node_alloc(void)
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_node_t *ret;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_lock(&base_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (base_nodes != NULL) {
ret = base_nodes;
base_nodes = *(extent_node_t **)ret;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_unlock(&base_mtx);
} else {
malloc_mutex_unlock(&base_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
ret = (extent_node_t *)base_alloc(sizeof(extent_node_t));
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
}
return (ret);
}
static void
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_node_dealloc(extent_node_t *node)
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
{
malloc_mutex_lock(&base_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
*(extent_node_t **)node = base_nodes;
base_nodes = node;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_unlock(&base_mtx);
}
/******************************************************************************/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
static void
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
stats_print(arena_t *arena)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
unsigned i, gap_start;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_printf("dirty: %zu page%s dirty, %llu sweep%s,"
" %llu madvise%s, %llu page%s purged\n",
arena->ndirty, arena->ndirty == 1 ? "" : "s",
arena->stats.npurge, arena->stats.npurge == 1 ? "" : "s",
arena->stats.nmadvise, arena->stats.nmadvise == 1 ? "" : "s",
arena->stats.purged, arena->stats.purged == 1 ? "" : "s");
malloc_printf(" allocated nmalloc ndalloc\n");
malloc_printf("small: %12zu %12llu %12llu\n",
arena->stats.allocated_small, arena->stats.nmalloc_small,
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena->stats.ndalloc_small);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_printf("large: %12zu %12llu %12llu\n",
arena->stats.allocated_large, arena->stats.nmalloc_large,
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena->stats.ndalloc_large);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_printf("total: %12zu %12llu %12llu\n",
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena->stats.allocated_small + arena->stats.allocated_large,
arena->stats.nmalloc_small + arena->stats.nmalloc_large,
arena->stats.ndalloc_small + arena->stats.ndalloc_large);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_printf("mapped: %12zu\n", arena->stats.mapped);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
if (__isthreaded && opt_mag) {
malloc_printf("bins: bin size regs pgs mags "
"newruns reruns maxruns curruns\n");
} else {
#endif
malloc_printf("bins: bin size regs pgs requests "
"newruns reruns maxruns curruns\n");
#ifdef MALLOC_MAG
}
#endif
for (i = 0, gap_start = UINT_MAX; i < nbins; i++) {
if (arena->bins[i].stats.nruns == 0) {
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
if (gap_start == UINT_MAX)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
gap_start = i;
} else {
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
if (gap_start != UINT_MAX) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (i > gap_start + 1) {
/* Gap of more than one size class. */
malloc_printf("[%u..%u]\n",
gap_start, i - 1);
} else {
/* Gap of one size class. */
malloc_printf("[%u]\n", gap_start);
}
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
gap_start = UINT_MAX;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
malloc_printf(
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
"%13u %1s %4u %4u %3u %9llu %9llu"
" %9llu %7lu %7lu\n",
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
i,
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
i < ntbins ? "T" : i < ntbins + nqbins ? "Q" :
i < ntbins + nqbins + ncbins ? "C" : "S",
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena->bins[i].reg_size,
arena->bins[i].nregs,
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
arena->bins[i].run_size >> PAGE_SHIFT,
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
(__isthreaded && opt_mag) ?
arena->bins[i].stats.nmags :
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena->bins[i].stats.nrequests,
arena->bins[i].stats.nruns,
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
arena->bins[i].stats.reruns,
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena->bins[i].stats.highruns,
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
arena->bins[i].stats.curruns);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
}
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
if (gap_start != UINT_MAX) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (i > gap_start + 1) {
/* Gap of more than one size class. */
malloc_printf("[%u..%u]\n", gap_start, i - 1);
} else {
/* Gap of one size class. */
malloc_printf("[%u]\n", gap_start);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
#endif
/*
* End Utility functions/macros.
*/
/******************************************************************************/
/*
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
* Begin extent tree code.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
#ifdef MALLOC_DSS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline int
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_szad_comp(extent_node_t *a, extent_node_t *b)
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
{
int ret;
size_t a_size = a->size;
size_t b_size = b->size;
ret = (a_size > b_size) - (a_size < b_size);
if (ret == 0) {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
uintptr_t a_addr = (uintptr_t)a->addr;
uintptr_t b_addr = (uintptr_t)b->addr;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
ret = (a_addr > b_addr) - (a_addr < b_addr);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
}
return (ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
/* Wrap red-black tree macros in functions. */
rb_wrap(__unused static, extent_tree_szad_, extent_tree_t, extent_node_t,
link_szad, extent_szad_comp)
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static inline int
extent_ad_comp(extent_node_t *a, extent_node_t *b)
{
uintptr_t a_addr = (uintptr_t)a->addr;
uintptr_t b_addr = (uintptr_t)b->addr;
return ((a_addr > b_addr) - (a_addr < b_addr));
}
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
/* Wrap red-black tree macros in functions. */
rb_wrap(__unused static, extent_tree_ad_, extent_tree_t, extent_node_t, link_ad,
extent_ad_comp)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/*
* End extent tree code.
*/
/******************************************************************************/
/*
* Begin chunk management functions.
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void *
pages_map(void *addr, size_t size)
{
void *ret;
/*
* We don't use MAP_FIXED here, because it can cause the *replacement*
* of existing mappings, and we only want to create new mappings.
*/
ret = mmap(addr, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON,
-1, 0);
assert(ret != NULL);
if (ret == MAP_FAILED)
ret = NULL;
else if (addr != NULL && ret != addr) {
/*
* We succeeded in mapping memory, but not in the right place.
*/
if (munmap(ret, size) == -1) {
char buf[STRERROR_BUF];
strerror_r(errno, buf, sizeof(buf));
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in munmap(): ", buf, "\n");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (opt_abort)
abort();
}
ret = NULL;
}
assert(ret == NULL || (addr == NULL && ret != addr)
|| (addr != NULL && ret == addr));
return (ret);
}
static void
pages_unmap(void *addr, size_t size)
{
if (munmap(addr, size) == -1) {
char buf[STRERROR_BUF];
strerror_r(errno, buf, sizeof(buf));
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in munmap(): ", buf, "\n");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (opt_abort)
abort();
}
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void *
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
chunk_alloc_dss(size_t size)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Check for integer overflow before calling sbrk(2), since it uses a signed increment argument, but the size is an unsigned integer.
2008-04-29 01:32:42 +00:00
/*
* sbrk() uses a signed increment argument, so take care not to
* interpret a huge allocation request as a negative increment.
*/
if ((intptr_t)size < 0)
return (NULL);
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_lock(&dss_mtx);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
if (dss_prev != (void *)-1) {
intptr_t incr;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
* The loop is necessary to recover from races with other
* threads that are using the DSS for something other than
* malloc.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
do {
void *ret;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
/* Get the current end of the DSS. */
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
dss_max = sbrk(0);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
/*
* Calculate how much padding is necessary to
* chunk-align the end of the DSS.
*/
incr = (intptr_t)size
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
- (intptr_t)CHUNK_ADDR2OFFSET(dss_max);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (incr == (intptr_t)size)
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
ret = dss_max;
else {
ret = (void *)((intptr_t)dss_max + incr);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
incr += size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
dss_prev = sbrk(incr);
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
if (dss_prev == dss_max) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Success. */
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
dss_max = (void *)((intptr_t)dss_prev + incr);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
malloc_mutex_unlock(&dss_mtx);
return (ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
} while (dss_prev != (void *)-1);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
}
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
return (NULL);
}
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void *
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
chunk_recycle_dss(size_t size, bool zero)
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_node_t *node, key;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
key.addr = NULL;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
key.size = size;
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_lock(&dss_mtx);
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
node = extent_tree_szad_nsearch(&dss_chunks_szad, &key);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (node != NULL) {
void *ret = node->addr;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
/* Remove node from the tree. */
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_remove(&dss_chunks_szad, node);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
if (node->size == size) {
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_ad_remove(&dss_chunks_ad, node);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_node_dealloc(node);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
} else {
/*
* Insert the remainder of node's address range as a
* smaller chunk. Its position within dss_chunks_ad
* does not change.
*/
assert(node->size > size);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
node->addr = (void *)((uintptr_t)node->addr + size);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
node->size -= size;
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_insert(&dss_chunks_szad, node);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
}
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
if (zero)
memset(ret, 0, size);
return (ret);
}
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
return (NULL);
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void *
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
chunk_alloc_mmap(size_t size)
{
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
void *ret;
size_t offset;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
/*
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
* Ideally, there would be a way to specify alignment to mmap() (like
* NetBSD has), but in the absence of such a feature, we have to work
* hard to efficiently create aligned mappings. The reliable, but
* expensive method is to create a mapping that is over-sized, then
* trim the excess. However, that always results in at least one call
* to pages_unmap().
*
* A more optimistic approach is to try mapping precisely the right
* amount, then try to append another mapping if alignment is off. In
* practice, this works out well as long as the application is not
* interleaving mappings via direct mmap() calls. If we do run into a
* situation where there is an interleaved mapping and we are unable to
* extend an unaligned mapping, our best option is to momentarily
* revert to the reliable-but-expensive method. This will tend to
* leave a gap in the memory map that is too small to cause later
* problems for the optimistic method.
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
*/
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
ret = pages_map(NULL, size);
if (ret == NULL)
return (NULL);
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
offset = CHUNK_ADDR2OFFSET(ret);
if (offset != 0) {
/* Try to extend chunk boundary. */
if (pages_map((void *)((uintptr_t)ret + size),
chunksize - offset) == NULL) {
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
/*
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
* Extension failed. Clean up, then revert to the
* reliable-but-expensive method.
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
*/
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
pages_unmap(ret, size);
/* Beware size_t wrap-around. */
if (size + chunksize <= size)
return NULL;
ret = pages_map(NULL, size + chunksize);
if (ret == NULL)
return (NULL);
/* Clean up unneeded leading/trailing space. */
Enable both sbrk(2)- and mmap(2)-based memory acquisition methods by default. This has the disadvantage of rendering the datasize resource limit irrelevant, but without this change, legitimate uses of more memory than will fit in the data segment are thwarted by default. Fix chunk_alloc_mmap() to work correctly if initial mapping is not chunk-aligned and mapping extension fails.
2008-01-03 23:22:13 +00:00
offset = CHUNK_ADDR2OFFSET(ret);
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
if (offset != 0) {
/* Leading space. */
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
pages_unmap(ret, chunksize - offset);
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
ret = (void *)((uintptr_t)ret +
(chunksize - offset));
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
/* Trailing space. */
pages_unmap((void *)((uintptr_t)ret + size),
offset);
} else {
/* Trailing space only. */
pages_unmap((void *)((uintptr_t)ret + size),
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunksize);
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
}
Enable both sbrk(2)- and mmap(2)-based memory acquisition methods by default. This has the disadvantage of rendering the datasize resource limit irrelevant, but without this change, legitimate uses of more memory than will fit in the data segment are thwarted by default. Fix chunk_alloc_mmap() to work correctly if initial mapping is not chunk-aligned and mapping extension fails.
2008-01-03 23:22:13 +00:00
} else {
/* Clean up unneeded leading space. */
pages_unmap(ret, chunksize - offset);
ret = (void *)((uintptr_t)ret + (chunksize - offset));
Modify chunk_alloc() to prefer mmap()ed memory over sbrk()ed memory. This has no impact unless USE_BRK is defined (32-bit platforms), in which case user allocations are allocated via mmap() if at all possible, in order to avoid the possibility of unreclaimable chunks in the data segment. Fix an obscure bug in base_alloc() that could have allowed undefined behavior if an application were to use sbrk() in conjunction with a USE_BRK-enabled malloc.
2007-02-22 19:10:30 +00:00
}
}
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
return (ret);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
}
static void *
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
chunk_alloc(size_t size, bool zero)
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
{
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
void *ret;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
assert(size != 0);
assert((size & chunksize_mask) == 0);
Revert to preferring mmap(2) over sbrk(2) when mapping memory, due to potential extreme contention in the kernel for multi-threaded applications on SMP systems. Reported by: kris
2008-11-03 21:17:18 +00:00
#ifdef MALLOC_DSS
if (opt_mmap)
#endif
{
ret = chunk_alloc_mmap(size);
if (ret != NULL)
goto RETURN;
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
if (opt_dss) {
ret = chunk_recycle_dss(size, zero);
if (ret != NULL) {
goto RETURN;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
}
ret = chunk_alloc_dss(size);
if (ret != NULL)
goto RETURN;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* All strategies for allocation failed. */
ret = NULL;
RETURN:
#ifdef MALLOC_STATS
if (ret != NULL) {
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
stats_chunks.nchunks += (size / chunksize);
stats_chunks.curchunks += (size / chunksize);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
if (stats_chunks.curchunks > stats_chunks.highchunks)
stats_chunks.highchunks = stats_chunks.curchunks;
#endif
assert(CHUNK_ADDR2BASE(ret) == ret);
return (ret);
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static extent_node_t *
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
chunk_dealloc_dss_record(void *chunk, size_t size)
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_node_t *node, *prev, key;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
key.addr = (void *)((uintptr_t)chunk + size);
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
node = extent_tree_ad_nsearch(&dss_chunks_ad, &key);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
/* Try to coalesce forward. */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (node != NULL && node->addr == key.addr) {
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
/*
* Coalesce chunk with the following address range. This does
* not change the position within dss_chunks_ad, so only
* remove/insert from/into dss_chunks_szad.
*/
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_remove(&dss_chunks_szad, node);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
node->addr = chunk;
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
node->size += size;
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_insert(&dss_chunks_szad, node);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
} else {
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
/*
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
* Coalescing forward failed, so insert a new node. Drop
* dss_mtx during node allocation, since it is possible that a
* new base chunk will be allocated.
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
*/
malloc_mutex_unlock(&dss_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
node = base_node_alloc();
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_lock(&dss_mtx);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
if (node == NULL)
return (NULL);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
node->addr = chunk;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
node->size = size;
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_ad_insert(&dss_chunks_ad, node);
extent_tree_szad_insert(&dss_chunks_szad, node);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
}
/* Try to coalesce backward. */
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
prev = extent_tree_ad_prev(&dss_chunks_ad, node);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (prev != NULL && (void *)((uintptr_t)prev->addr + prev->size) ==
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
chunk) {
/*
* Coalesce chunk with the previous address range. This does
* not change the position within dss_chunks_ad, so only
* remove/insert node from/into dss_chunks_szad.
*/
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_remove(&dss_chunks_szad, prev);
extent_tree_ad_remove(&dss_chunks_ad, prev);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_remove(&dss_chunks_szad, node);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
node->addr = prev->addr;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
node->size += prev->size;
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_insert(&dss_chunks_szad, node);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_node_dealloc(prev);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
}
return (node);
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static bool
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
chunk_dealloc_dss(void *chunk, size_t size)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_lock(&dss_mtx);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
if ((uintptr_t)chunk >= (uintptr_t)dss_base
&& (uintptr_t)chunk < (uintptr_t)dss_max) {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_node_t *node;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
/* Try to coalesce with other unused chunks. */
node = chunk_dealloc_dss_record(chunk, size);
if (node != NULL) {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
chunk = node->addr;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
size = node->size;
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
/* Get the current end of the DSS. */
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
dss_max = sbrk(0);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
/*
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
* Try to shrink the DSS if this chunk is at the end of the
* DSS. The sbrk() call here is subject to a race condition
* with threads that use brk(2) or sbrk(2) directly, but the
* alternative would be to leak memory for the sake of poorly
* designed multi-threaded programs.
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
*/
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
if ((void *)((uintptr_t)chunk + size) == dss_max
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
&& (dss_prev = sbrk(-(intptr_t)size)) == dss_max) {
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
/* Success. */
dss_max = (void *)((intptr_t)dss_prev - (intptr_t)size);
if (node != NULL) {
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_szad_remove(&dss_chunks_szad, node);
extent_tree_ad_remove(&dss_chunks_ad, node);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_node_dealloc(node);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
}
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
} else {
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
malloc_mutex_unlock(&dss_mtx);
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
madvise(chunk, size, MADV_FREE);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
}
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
return (false);
}
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
return (true);
}
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
chunk_dealloc_mmap(void *chunk, size_t size)
{
pages_unmap(chunk, size);
}
static void
chunk_dealloc(void *chunk, size_t size)
{
assert(chunk != NULL);
assert(CHUNK_ADDR2BASE(chunk) == chunk);
assert(size != 0);
assert((size & chunksize_mask) == 0);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
#ifdef MALLOC_STATS
stats_chunks.curchunks -= (size / chunksize);
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
if (opt_dss) {
if (chunk_dealloc_dss(chunk, size) == false)
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
return;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
}
if (opt_mmap)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
chunk_dealloc_mmap(chunk, size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* End chunk management functions.
*/
/******************************************************************************/
/*
* Begin arena.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
/*
* Choose an arena based on a per-thread value (fast-path code, calls slow-path
Fix junk/zero filling for realloc(). Junk filling was missing in one case, and zero filling was broken in a way that could cause memory corruption. Update comments.
2007-06-15 22:00:16 +00:00
* code if necessary).
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
*/
static inline arena_t *
choose_arena(void)
{
arena_t *ret;
/*
* We can only use TLS if this is a PIC library, since for the static
* library version, libc's malloc is used by TLS allocation, which
* introduces a bootstrapping issue.
*/
#ifndef NO_TLS
if (__isthreaded == false) {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Avoid the overhead of TLS for single-threaded operation. */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
return (arenas[0]);
}
ret = arenas_map;
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
if (ret == NULL) {
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = choose_arena_hard();
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
assert(ret != NULL);
}
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#else
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (__isthreaded && narenas > 1) {
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
unsigned long ind;
/*
* Hash _pthread_self() to one of the arenas. There is a prime
* number of arenas, so this has a reasonable chance of
* working. Even so, the hashing can be easily thwarted by
* inconvenient _pthread_self() values. Without specific
* knowledge of how _pthread_self() calculates values, we can't
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
* easily do much better than this.
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
*/
ind = (unsigned long) _pthread_self() % narenas;
/*
* Optimistially assume that arenas[ind] has been initialized.
* At worst, we find out that some other thread has already
* done so, after acquiring the lock in preparation. Note that
* this lazy locking also has the effect of lazily forcing
* cache coherency; without the lock acquisition, there's no
* guarantee that modification of arenas[ind] by another thread
* would be seen on this CPU for an arbitrary amount of time.
*
* In general, this approach to modifying a synchronized value
* isn't a good idea, but in this case we only ever modify the
* value once, so things work out well.
*/
ret = arenas[ind];
if (ret == NULL) {
/*
* Avoid races with another thread that may have already
* initialized arenas[ind].
*/
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_lock(&arenas_lock);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
if (arenas[ind] == NULL)
ret = arenas_extend((unsigned)ind);
else
ret = arenas[ind];
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_unlock(&arenas_lock);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
}
} else
ret = arenas[0];
#endif
assert(ret != NULL);
return (ret);
}
#ifndef NO_TLS
/*
* Choose an arena based on a per-thread value (slow-path code only, called
* only by choose_arena()).
*/
static arena_t *
choose_arena_hard(void)
{
arena_t *ret;
assert(__isthreaded);
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Remove support for lazy deallocation. Benchmarks across a wide range of allocation patterns, number of CPUs, and MALLOC_OPTIONS settings indicate that lazy deallocation has the potential to worsen throughput dramatically. Performance degradation occurs when multiple threads try to clear the lazy free cache simultaneously. Various experiments to avoid this bottleneck failed to completely solve this problem, while adding yet more complexity.
2008-02-17 17:09:24 +00:00
/* Seed the PRNG used for arena load balancing. */
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
SPRN(balance, (uint32_t)(uintptr_t)(_pthread_self()));
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#endif
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
if (narenas > 1) {
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
unsigned ind;
ind = PRN(balance, narenas_2pow);
if ((ret = arenas[ind]) == NULL) {
malloc_spin_lock(&arenas_lock);
if ((ret = arenas[ind]) == NULL)
ret = arenas_extend(ind);
malloc_spin_unlock(&arenas_lock);
}
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#else
malloc_spin_lock(&arenas_lock);
if ((ret = arenas[next_arena]) == NULL)
ret = arenas_extend(next_arena);
next_arena = (next_arena + 1) % narenas;
malloc_spin_unlock(&arenas_lock);
#endif
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
} else
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = arenas[0];
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arenas_map = ret;
return (ret);
}
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline int
arena_chunk_comp(arena_chunk_t *a, arena_chunk_t *b)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
uintptr_t a_chunk = (uintptr_t)a;
uintptr_t b_chunk = (uintptr_t)b;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(a != NULL);
assert(b != NULL);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
return ((a_chunk > b_chunk) - (a_chunk < b_chunk));
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
/* Wrap red-black tree macros in functions. */
rb_wrap(__unused static, arena_chunk_tree_dirty_, arena_chunk_tree_t,
arena_chunk_t, link_dirty, arena_chunk_comp)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
static inline int
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_comp(arena_chunk_map_t *a, arena_chunk_map_t *b)
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
{
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
uintptr_t a_mapelm = (uintptr_t)a;
uintptr_t b_mapelm = (uintptr_t)b;
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
assert(a != NULL);
assert(b != NULL);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
return ((a_mapelm > b_mapelm) - (a_mapelm < b_mapelm));
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
}
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
/* Wrap red-black tree macros in functions. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
rb_wrap(__unused static, arena_run_tree_, arena_run_tree_t, arena_chunk_map_t,
link, arena_run_comp)
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
static inline int
arena_avail_comp(arena_chunk_map_t *a, arena_chunk_map_t *b)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
{
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
int ret;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
size_t a_size = a->bits & ~PAGE_MASK;
size_t b_size = b->bits & ~PAGE_MASK;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
ret = (a_size > b_size) - (a_size < b_size);
if (ret == 0) {
uintptr_t a_mapelm, b_mapelm;
if ((a->bits & CHUNK_MAP_KEY) == 0)
a_mapelm = (uintptr_t)a;
else {
/*
* Treat keys as though they are lower than anything
* else.
*/
a_mapelm = 0;
}
b_mapelm = (uintptr_t)b;
ret = (a_mapelm > b_mapelm) - (a_mapelm < b_mapelm);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
return (ret);
}
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Wrap red-black tree macros in functions. */
rb_wrap(__unused static, arena_avail_tree_, arena_avail_tree_t,
arena_chunk_map_t, link, arena_avail_comp)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
static inline void *
arena_run_reg_alloc(arena_run_t *run, arena_bin_t *bin)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
void *ret;
unsigned i, mask, bit, regind;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(run->magic == ARENA_RUN_MAGIC);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
assert(run->regs_minelm < bin->regs_mask_nelms);
/*
* Move the first check outside the loop, so that run->regs_minelm can
* be updated unconditionally, without the possibility of updating it
* multiple times.
*/
i = run->regs_minelm;
mask = run->regs_mask[i];
if (mask != 0) {
/* Usable allocation found. */
bit = ffs((int)mask) - 1;
regind = ((i << (SIZEOF_INT_2POW + 3)) + bit);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(regind < bin->nregs);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
ret = (void *)(((uintptr_t)run) + bin->reg0_offset
+ (bin->reg_size * regind));
/* Clear bit. */
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
mask ^= (1U << bit);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
run->regs_mask[i] = mask;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
return (ret);
}
for (i++; i < bin->regs_mask_nelms; i++) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
mask = run->regs_mask[i];
if (mask != 0) {
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/* Usable allocation found. */
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
bit = ffs((int)mask) - 1;
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
regind = ((i << (SIZEOF_INT_2POW + 3)) + bit);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(regind < bin->nregs);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
ret = (void *)(((uintptr_t)run) + bin->reg0_offset
+ (bin->reg_size * regind));
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/* Clear bit. */
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
mask ^= (1U << bit);
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
run->regs_mask[i] = mask;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Make a note that nothing before this element
* contains a free region.
*/
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
run->regs_minelm = i; /* Low payoff: + (mask == 0); */
return (ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/* Not reached. */
assert(0);
Add an unreachable return statement, in order to avoid a compiler warning for non-standard optimization levels. Reported by: Michael Zach <zach@webges.com>
2006-04-05 18:46:24 +00:00
return (NULL);
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
}
static inline void
arena_run_reg_dalloc(arena_run_t *run, arena_bin_t *bin, void *ptr, size_t size)
{
unsigned diff, regind, elm, bit;
assert(run->magic == ARENA_RUN_MAGIC);
/*
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
* Avoid doing division with a variable divisor if possible. Using
* actual division here can reduce allocator throughput by over 20%!
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
*/
diff = (unsigned)((uintptr_t)ptr - (uintptr_t)run - bin->reg0_offset);
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
if ((size & (size - 1)) == 0) {
/*
* log2_table allows fast division of a power of two in the
* [1..128] range.
*
* (x / divisor) becomes (x >> log2_table[divisor - 1]).
*/
static const unsigned char log2_table[] = {
0, 1, 0, 2, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7
};
if (size <= 128)
regind = (diff >> log2_table[size - 1]);
else if (size <= 32768)
regind = diff >> (8 + log2_table[(size >> 8) - 1]);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
else
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
regind = diff / size;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
} else if (size < qspace_max) {
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
/*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* To divide by a number D that is not a power of two we
* multiply by (2^21 / D) and then right shift by 21 positions.
*
* X / D
*
* becomes
*
* (X * qsize_invs[(D >> QUANTUM_2POW) - 3])
* >> SIZE_INV_SHIFT
*
* We can omit the first three elements, because we never
* divide by 0, and QUANTUM and 2*QUANTUM are both powers of
* two, which are handled above.
Use some math tricks in arena_run_reg_dalloc() to avoid actual division, as well as avoiding a switch statement. This change has no significant impact to performance when branch prediction is successful at predicting the sizes of objects passed to free(), but in the case that the object sizes are semi-random, this change has the potential to prevent many branch prediction misses, thus improving performance substantially. Take advantage of alignment guarantees in ipalloc(), and pad object sizes to something less than a power of two when possible. This has the potential to substantially reduce internal fragmentation for objects allocated via posix_memalign(). Avoid an unnecessary pow2_ceil() call in arena_ralloc(). Submitted by: djam8193ah@hotmail.com
2006-07-01 16:51:10 +00:00
*/
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#define SIZE_INV_SHIFT 21
#define QSIZE_INV(s) (((1U << SIZE_INV_SHIFT) / (s << QUANTUM_2POW)) + 1)
static const unsigned qsize_invs[] = {
QSIZE_INV(3),
QSIZE_INV(4), QSIZE_INV(5), QSIZE_INV(6), QSIZE_INV(7)
#if (QUANTUM_2POW < 4)
,
QSIZE_INV(8), QSIZE_INV(9), QSIZE_INV(10), QSIZE_INV(11),
QSIZE_INV(12),QSIZE_INV(13), QSIZE_INV(14), QSIZE_INV(15)
#endif
};
assert(QUANTUM * (((sizeof(qsize_invs)) / sizeof(unsigned)) + 3)
>= (1U << QSPACE_MAX_2POW_DEFAULT));
if (size <= (((sizeof(qsize_invs) / sizeof(unsigned)) + 2) <<
QUANTUM_2POW)) {
regind = qsize_invs[(size >> QUANTUM_2POW) - 3] * diff;
regind >>= SIZE_INV_SHIFT;
} else
regind = diff / size;
#undef QSIZE_INV
} else if (size < cspace_max) {
#define CSIZE_INV(s) (((1U << SIZE_INV_SHIFT) / (s << CACHELINE_2POW)) + 1)
static const unsigned csize_invs[] = {
CSIZE_INV(3),
CSIZE_INV(4), CSIZE_INV(5), CSIZE_INV(6), CSIZE_INV(7)
};
assert(CACHELINE * (((sizeof(csize_invs)) / sizeof(unsigned)) +
3) >= (1U << CSPACE_MAX_2POW_DEFAULT));
if (size <= (((sizeof(csize_invs) / sizeof(unsigned)) + 2) <<
CACHELINE_2POW)) {
regind = csize_invs[(size >> CACHELINE_2POW) - 3] *
diff;
regind >>= SIZE_INV_SHIFT;
} else
regind = diff / size;
#undef CSIZE_INV
} else {
#define SSIZE_INV(s) (((1U << SIZE_INV_SHIFT) / (s << SUBPAGE_2POW)) + 1)
static const unsigned ssize_invs[] = {
SSIZE_INV(3),
SSIZE_INV(4), SSIZE_INV(5), SSIZE_INV(6), SSIZE_INV(7),
SSIZE_INV(8), SSIZE_INV(9), SSIZE_INV(10), SSIZE_INV(11),
SSIZE_INV(12), SSIZE_INV(13), SSIZE_INV(14), SSIZE_INV(15)
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
#if (PAGE_SHIFT == 13)
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
,
SSIZE_INV(16), SSIZE_INV(17), SSIZE_INV(18), SSIZE_INV(19),
SSIZE_INV(20), SSIZE_INV(21), SSIZE_INV(22), SSIZE_INV(23),
SSIZE_INV(24), SSIZE_INV(25), SSIZE_INV(26), SSIZE_INV(27),
SSIZE_INV(28), SSIZE_INV(29), SSIZE_INV(29), SSIZE_INV(30)
#endif
};
assert(SUBPAGE * (((sizeof(ssize_invs)) / sizeof(unsigned)) + 3)
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
>= PAGE_SIZE);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
if (size < (((sizeof(ssize_invs) / sizeof(unsigned)) + 2) <<
SUBPAGE_2POW)) {
regind = ssize_invs[(size >> SUBPAGE_2POW) - 3] * diff;
regind >>= SIZE_INV_SHIFT;
} else
regind = diff / size;
#undef SSIZE_INV
}
#undef SIZE_INV_SHIFT
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
assert(diff == regind * size);
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
assert(regind < bin->nregs);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
elm = regind >> (SIZEOF_INT_2POW + 3);
if (elm < run->regs_minelm)
run->regs_minelm = elm;
bit = regind - (elm << (SIZEOF_INT_2POW + 3));
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
assert((run->regs_mask[elm] & (1U << bit)) == 0);
run->regs_mask[elm] |= (1U << bit);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_split(arena_t *arena, arena_run_t *run, size_t size, bool large,
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
bool zero)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
size_t old_ndirty, run_ind, total_pages, need_pages, rem_pages, i;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(run);
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
old_ndirty = chunk->ndirty;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run_ind = (unsigned)(((uintptr_t)run - (uintptr_t)chunk)
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
>> PAGE_SHIFT);
total_pages = (chunk->map[run_ind].bits & ~PAGE_MASK) >>
PAGE_SHIFT;
need_pages = (size >> PAGE_SHIFT);
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
assert(need_pages > 0);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
assert(need_pages <= total_pages);
rem_pages = total_pages - need_pages;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_avail_tree_remove(&arena->runs_avail, &chunk->map[run_ind]);
/* Keep track of trailing unused pages for later use. */
if (rem_pages > 0) {
chunk->map[run_ind+need_pages].bits = (rem_pages <<
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_SHIFT) | (chunk->map[run_ind+need_pages].bits &
PAGE_MASK);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[run_ind+total_pages-1].bits = (rem_pages <<
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_SHIFT) | (chunk->map[run_ind+total_pages-1].bits &
PAGE_MASK);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_avail_tree_insert(&arena->runs_avail,
&chunk->map[run_ind+need_pages]);
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
for (i = 0; i < need_pages; i++) {
/* Zero if necessary. */
if (zero) {
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if ((chunk->map[run_ind + i].bits & CHUNK_MAP_ZEROED)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
== 0) {
memset((void *)((uintptr_t)chunk + ((run_ind
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
+ i) << PAGE_SHIFT)), 0, PAGE_SIZE);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* CHUNK_MAP_ZEROED is cleared below. */
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
}
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Update dirty page accounting. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if (chunk->map[run_ind + i].bits & CHUNK_MAP_DIRTY) {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
chunk->ndirty--;
arena->ndirty--;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* CHUNK_MAP_DIRTY is cleared below. */
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Initialize the chunk map. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if (large) {
chunk->map[run_ind + i].bits = CHUNK_MAP_LARGE
| CHUNK_MAP_ALLOCATED;
} else {
chunk->map[run_ind + i].bits = (size_t)run
| CHUNK_MAP_ALLOCATED;
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/*
* Set the run size only in the first element for large runs. This is
* primarily a debugging aid, since the lack of size info for trailing
* pages only matters if the application tries to operate on an
* interior pointer.
*/
if (large)
chunk->map[run_ind].bits |= size;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
if (chunk->ndirty == 0 && old_ndirty > 0)
arena_chunk_tree_dirty_remove(&arena->chunks_dirty, chunk);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_chunk_t *
arena_chunk_alloc(arena_t *arena)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
size_t i;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
if (arena->spare != NULL) {
chunk = arena->spare;
arena->spare = NULL;
} else {
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
chunk = (arena_chunk_t *)chunk_alloc(chunksize, true);
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
if (chunk == NULL)
return (NULL);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifdef MALLOC_STATS
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena->stats.mapped += chunksize;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
chunk->arena = arena;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
/*
* Claim that no pages are in use, since the header is merely
* overhead.
*/
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
chunk->ndirty = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
/*
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
* Initialize the map to contain one maximal free untouched run.
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
for (i = 0; i < arena_chunk_header_npages; i++)
chunk->map[i].bits = 0;
chunk->map[i].bits = arena_maxclass | CHUNK_MAP_ZEROED;
for (i++; i < chunk_npages-1; i++) {
chunk->map[i].bits = CHUNK_MAP_ZEROED;
}
chunk->map[chunk_npages-1].bits = arena_maxclass |
CHUNK_MAP_ZEROED;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Insert the run into the runs_avail tree. */
arena_avail_tree_insert(&arena->runs_avail,
&chunk->map[arena_chunk_header_npages]);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (chunk);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
arena_chunk_dealloc(arena_t *arena, arena_chunk_t *chunk)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (arena->spare != NULL) {
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
if (arena->spare->ndirty > 0) {
arena_chunk_tree_dirty_remove(
&chunk->arena->chunks_dirty, arena->spare);
arena->ndirty -= arena->spare->ndirty;
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
chunk_dealloc((void *)arena->spare, chunksize);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#ifdef MALLOC_STATS
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena->stats.mapped -= chunksize;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
#endif
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/*
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
* Remove run from runs_avail, regardless of whether this chunk
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
* will be cached, so that the arena does not use it. Dirty page
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
* flushing only uses the chunks_dirty tree, so leaving this chunk in
* the chunks_* trees is sufficient for that purpose.
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_avail_tree_remove(&arena->runs_avail,
&chunk->map[arena_chunk_header_npages]);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena->spare = chunk;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_run_t *
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_alloc(arena_t *arena, size_t size, bool large, bool zero)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
arena_chunk_t *chunk;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
arena_run_t *run;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_chunk_map_t *mapelm, key;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
assert(size <= arena_maxclass);
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert((size & PAGE_MASK) == 0);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Search the arena's chunks for the lowest best fit. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
key.bits = size | CHUNK_MAP_KEY;
mapelm = arena_avail_tree_nsearch(&arena->runs_avail, &key);
if (mapelm != NULL) {
arena_chunk_t *run_chunk = CHUNK_ADDR2BASE(mapelm);
size_t pageind = ((uintptr_t)mapelm - (uintptr_t)run_chunk->map)
/ sizeof(arena_chunk_map_t);
run = (arena_run_t *)((uintptr_t)run_chunk + (pageind
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
<< PAGE_SHIFT));
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_split(arena, run, size, large, zero);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
return (run);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Fix junk/zero filling for realloc(). Junk filling was missing in one case, and zero filling was broken in a way that could cause memory corruption. Update comments.
2007-06-15 22:00:16 +00:00
/*
* No usable runs. Create a new chunk from which to allocate the run.
*/
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
chunk = arena_chunk_alloc(arena);
if (chunk == NULL)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (NULL);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
run = (arena_run_t *)((uintptr_t)chunk + (arena_chunk_header_npages <<
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_SHIFT));
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Update page map. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_split(arena, run, size, large, zero);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
return (run);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static void
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_purge(arena_t *arena)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_chunk_t *chunk;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
size_t i, npages;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#ifdef MALLOC_DEBUG
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
size_t ndirty = 0;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
rb_foreach_begin(arena_chunk_t, link_dirty, &arena->chunks_dirty,
chunk) {
ndirty += chunk->ndirty;
} rb_foreach_end(arena_chunk_t, link_dirty, &arena->chunks_dirty, chunk)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(ndirty == arena->ndirty);
#endif
assert(arena->ndirty > opt_dirty_max);
#ifdef MALLOC_STATS
arena->stats.npurge++;
#endif
/*
* Iterate downward through chunks until enough dirty memory has been
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
* purged. Terminate as soon as possible in order to minimize the
* number of system calls, even if a chunk has only been partially
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
* purged.
*/
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
while (arena->ndirty > (opt_dirty_max >> 1)) {
chunk = arena_chunk_tree_dirty_last(&arena->chunks_dirty);
assert(chunk != NULL);
for (i = chunk_npages - 1; chunk->ndirty > 0; i--) {
assert(i >= arena_chunk_header_npages);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if (chunk->map[i].bits & CHUNK_MAP_DIRTY) {
chunk->map[i].bits ^= CHUNK_MAP_DIRTY;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
/* Find adjacent dirty run(s). */
for (npages = 1; i > arena_chunk_header_npages
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
&& (chunk->map[i - 1].bits &
CHUNK_MAP_DIRTY); npages++) {
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
i--;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[i].bits ^= CHUNK_MAP_DIRTY;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
}
chunk->ndirty -= npages;
arena->ndirty -= npages;
madvise((void *)((uintptr_t)chunk + (i <<
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_SHIFT)), (npages << PAGE_SHIFT),
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
MADV_FREE);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#ifdef MALLOC_STATS
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
arena->stats.nmadvise++;
arena->stats.purged += npages;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#endif
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
if (arena->ndirty <= (opt_dirty_max >> 1))
break;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
}
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
if (chunk->ndirty == 0) {
arena_chunk_tree_dirty_remove(&arena->chunks_dirty,
chunk);
}
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
static void
arena_run_dalloc(arena_t *arena, arena_run_t *run, bool dirty)
{
arena_chunk_t *chunk;
size_t size, run_ind, run_pages;
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(run);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
run_ind = (size_t)(((uintptr_t)run - (uintptr_t)chunk)
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
>> PAGE_SHIFT);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert(run_ind >= arena_chunk_header_npages);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
assert(run_ind < chunk_npages);
if ((chunk->map[run_ind].bits & CHUNK_MAP_LARGE) != 0)
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
size = chunk->map[run_ind].bits & ~PAGE_MASK;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
else
size = run->bin->run_size;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run_pages = (size >> PAGE_SHIFT);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Mark pages as unallocated in the chunk map. */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (dirty) {
size_t i;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
for (i = 0; i < run_pages; i++) {
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
assert((chunk->map[run_ind + i].bits & CHUNK_MAP_DIRTY)
== 0);
chunk->map[run_ind + i].bits = CHUNK_MAP_DIRTY;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
if (chunk->ndirty == 0) {
arena_chunk_tree_dirty_insert(&arena->chunks_dirty,
chunk);
}
chunk->ndirty += run_pages;
arena->ndirty += run_pages;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
} else {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
size_t i;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
for (i = 0; i < run_pages; i++) {
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[run_ind + i].bits &= ~(CHUNK_MAP_LARGE |
CHUNK_MAP_ALLOCATED);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
}
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[run_ind].bits = size | (chunk->map[run_ind].bits &
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_MASK);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[run_ind+run_pages-1].bits = size |
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(chunk->map[run_ind+run_pages-1].bits & PAGE_MASK);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Try to coalesce forward. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if (run_ind + run_pages < chunk_npages &&
(chunk->map[run_ind+run_pages].bits & CHUNK_MAP_ALLOCATED) == 0) {
size_t nrun_size = chunk->map[run_ind+run_pages].bits &
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
~PAGE_MASK;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
/*
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
* Remove successor from runs_avail; the coalesced run is
* inserted later.
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_avail_tree_remove(&arena->runs_avail,
&chunk->map[run_ind+run_pages]);
size += nrun_size;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run_pages = size >> PAGE_SHIFT;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert((chunk->map[run_ind+run_pages-1].bits & ~PAGE_MASK)
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
== nrun_size);
chunk->map[run_ind].bits = size | (chunk->map[run_ind].bits &
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_MASK);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[run_ind+run_pages-1].bits = size |
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(chunk->map[run_ind+run_pages-1].bits & PAGE_MASK);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Try to coalesce backward. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if (run_ind > arena_chunk_header_npages && (chunk->map[run_ind-1].bits &
CHUNK_MAP_ALLOCATED) == 0) {
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
size_t prun_size = chunk->map[run_ind-1].bits & ~PAGE_MASK;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run_ind -= prun_size >> PAGE_SHIFT;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/*
* Remove predecessor from runs_avail; the coalesced run is
* inserted later.
*/
arena_avail_tree_remove(&arena->runs_avail,
&chunk->map[run_ind]);
size += prun_size;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run_pages = size >> PAGE_SHIFT;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert((chunk->map[run_ind].bits & ~PAGE_MASK) ==
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
prun_size);
chunk->map[run_ind].bits = size | (chunk->map[run_ind].bits &
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_MASK);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[run_ind+run_pages-1].bits = size |
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(chunk->map[run_ind+run_pages-1].bits & PAGE_MASK);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
/* Insert into runs_avail, now that coalescing is complete. */
arena_avail_tree_insert(&arena->runs_avail, &chunk->map[run_ind]);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Deallocate chunk if it is now completely unused. */
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
if ((chunk->map[arena_chunk_header_npages].bits & (~PAGE_MASK |
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
CHUNK_MAP_ALLOCATED)) == arena_maxclass)
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
arena_chunk_dealloc(arena, chunk);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Enforce opt_dirty_max. */
if (arena->ndirty > opt_dirty_max)
arena_purge(arena);
}
static void
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_trim_head(arena_t *arena, arena_chunk_t *chunk, arena_run_t *run,
size_t oldsize, size_t newsize)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
{
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
size_t pageind = ((uintptr_t)run - (uintptr_t)chunk) >> PAGE_SHIFT;
size_t head_npages = (oldsize - newsize) >> PAGE_SHIFT;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(oldsize > newsize);
/*
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
* Update the chunk map so that arena_run_dalloc() can treat the
* leading run as separately allocated.
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[pageind].bits = (oldsize - newsize) | CHUNK_MAP_LARGE |
CHUNK_MAP_ALLOCATED;
chunk->map[pageind+head_npages].bits = newsize | CHUNK_MAP_LARGE |
CHUNK_MAP_ALLOCATED;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_dalloc(arena, run, false);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
static void
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_trim_tail(arena_t *arena, arena_chunk_t *chunk, arena_run_t *run,
size_t oldsize, size_t newsize, bool dirty)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
{
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
size_t pageind = ((uintptr_t)run - (uintptr_t)chunk) >> PAGE_SHIFT;
size_t npages = newsize >> PAGE_SHIFT;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(oldsize > newsize);
/*
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
* Update the chunk map so that arena_run_dalloc() can treat the
* trailing run as separately allocated.
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[pageind].bits = newsize | CHUNK_MAP_LARGE |
CHUNK_MAP_ALLOCATED;
chunk->map[pageind+npages].bits = (oldsize - newsize) | CHUNK_MAP_LARGE
| CHUNK_MAP_ALLOCATED;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_run_dalloc(arena, (arena_run_t *)((uintptr_t)run + newsize),
dirty);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static arena_run_t *
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_nonfull_run_get(arena_t *arena, arena_bin_t *bin)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_chunk_map_t *mapelm;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_run_t *run;
unsigned i, remainder;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Look for a usable run. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
mapelm = arena_run_tree_first(&bin->runs);
if (mapelm != NULL) {
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
/* run is guaranteed to have available space. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_tree_remove(&bin->runs, mapelm);
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run = (arena_run_t *)(mapelm->bits & ~PAGE_MASK);
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
#ifdef MALLOC_STATS
bin->stats.reruns++;
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (run);
}
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* No existing runs have any space available. */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Allocate a new run. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
run = arena_run_alloc(arena, bin->run_size, false, false);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (run == NULL)
return (NULL);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Initialize run internals. */
run->bin = bin;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
In the error path through base_alloc(), release base_mtx [1]. Fix bit vector initialization for run headers. Submitted by: [1] Mike Schuster <schuster@adobe.com>
2008-06-10 15:46:18 +00:00
for (i = 0; i < bin->regs_mask_nelms - 1; i++)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->regs_mask[i] = UINT_MAX;
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
remainder = bin->nregs & ((1U << (SIZEOF_INT_2POW + 3)) - 1);
In the error path through base_alloc(), release base_mtx [1]. Fix bit vector initialization for run headers. Submitted by: [1] Mike Schuster <schuster@adobe.com>
2008-06-10 15:46:18 +00:00
if (remainder == 0)
run->regs_mask[i] = UINT_MAX;
else {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/* The last element has spare bits that need to be unset. */
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
run->regs_mask[i] = (UINT_MAX >> ((1U << (SIZEOF_INT_2POW + 3))
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
- remainder));
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->regs_minelm = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->nfree = bin->nregs;
#ifdef MALLOC_DEBUG
run->magic = ARENA_RUN_MAGIC;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
bin->stats.nruns++;
bin->stats.curruns++;
if (bin->stats.curruns > bin->stats.highruns)
bin->stats.highruns = bin->stats.curruns;
Style changes. Inching closer to convergence with OpenBSD.
2003-10-25 12:56:51 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
return (run);
}
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* bin->runcur must have space available before this function is called. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline void *
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_malloc_easy(arena_t *arena, arena_bin_t *bin, arena_run_t *run)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
void *ret;
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(run->magic == ARENA_RUN_MAGIC);
Separate completely full runs from runs that are merely almost full, so that no linear searching is necessary if we resort to allocating from a run that is known to be mostly full. There are pathological edge cases that could have caused severely degraded performance, and this change fixes that.
2006-03-20 04:05:05 +00:00
assert(run->nfree > 0);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
ret = arena_run_reg_alloc(run, bin);
assert(ret != NULL);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
run->nfree--;
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
}
phkmalloc/3 Various neat features added. More documentation in the manpage. If your machine has very little RAM, I guess that would be < 16M these days :-(, you may want to try this: ln -fs 'H<' /etc/malloc.conf check the manpage.
1996-09-23 19:26:39 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Re-fill bin->runcur, then call arena_bin_malloc_easy(). */
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void *
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
arena_bin_malloc_hard(arena_t *arena, arena_bin_t *bin)
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
{
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
bin->runcur = arena_bin_nonfull_run_get(arena, bin);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
if (bin->runcur == NULL)
return (NULL);
assert(bin->runcur->magic == ARENA_RUN_MAGIC);
Convert TINY_MIN_2POW from a cpp macro to tiny_min_2pow (a variable), and determine its value at run time according to other relevant values. This avoids the creation of runs that are incompletely utilized, as long as pagesize isn't too large (>32kB, given the current RUN_MIN_REGS_2POW setting). Increase the size of several structure bitfields in arena_run_t in order to avoid integer overflow in the case that a run's header does not overlap with the space that is usable as application allocation regions. Given the tiny_min_2pow change, this fix has no additional impact unless pagesize is >32kB. Reported by: kris
2006-03-24 22:13:49 +00:00
assert(bin->runcur->nfree > 0);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Fix cpp logic in arena_malloc() to adjust size when assertions are enabled, even if stats gathering is disabled. [1] Remove 'size' parameter from several functions that do not use it. Reported by: [1] ache
2006-07-27 04:00:12 +00:00
return (arena_bin_malloc_easy(arena, bin, bin->runcur));
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
/*
* Calculate bin->run_size such that it meets the following constraints:
*
* *) bin->run_size >= min_run_size
* *) bin->run_size <= arena_maxclass
* *) bin->run_size <= RUN_MAX_SMALL
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
* *) run header overhead <= RUN_MAX_OVRHD (or header overhead relaxed).
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
*
* bin->nregs, bin->regs_mask_nelms, and bin->reg0_offset are
* also calculated here, since these settings are all interdependent.
*/
static size_t
arena_bin_run_size_calc(arena_bin_t *bin, size_t min_run_size)
{
size_t try_run_size, good_run_size;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
unsigned good_nregs, good_mask_nelms, good_reg0_offset;
unsigned try_nregs, try_mask_nelms, try_reg0_offset;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert(min_run_size >= PAGE_SIZE);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
assert(min_run_size <= arena_maxclass);
assert(min_run_size <= RUN_MAX_SMALL);
/*
* Calculate known-valid settings before entering the run_size
* expansion loop, so that the first part of the loop always copies
* valid settings.
*
* The do..while loop iteratively reduces the number of regions until
* the run header and the regions no longer overlap. A closed formula
* would be quite messy, since there is an interdependency between the
* header's mask length and the number of regions.
*/
try_run_size = min_run_size;
try_nregs = ((try_run_size - sizeof(arena_run_t)) / bin->reg_size)
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
+ 1; /* Counter-act try_nregs-- in loop. */
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
do {
try_nregs--;
try_mask_nelms = (try_nregs >> (SIZEOF_INT_2POW + 3)) +
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
((try_nregs & ((1U << (SIZEOF_INT_2POW + 3)) - 1)) ? 1 : 0);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
try_reg0_offset = try_run_size - (try_nregs * bin->reg_size);
} while (sizeof(arena_run_t) + (sizeof(unsigned) * (try_mask_nelms - 1))
> try_reg0_offset);
/* run_size expansion loop. */
do {
/*
* Copy valid settings before trying more aggressive settings.
*/
good_run_size = try_run_size;
good_nregs = try_nregs;
good_mask_nelms = try_mask_nelms;
good_reg0_offset = try_reg0_offset;
/* Try more aggressive settings. */
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
try_run_size += PAGE_SIZE;
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
try_nregs = ((try_run_size - sizeof(arena_run_t)) /
bin->reg_size) + 1; /* Counter-act try_nregs-- in loop. */
do {
try_nregs--;
try_mask_nelms = (try_nregs >> (SIZEOF_INT_2POW + 3)) +
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
((try_nregs & ((1U << (SIZEOF_INT_2POW + 3)) - 1)) ?
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
1 : 0);
try_reg0_offset = try_run_size - (try_nregs *
bin->reg_size);
} while (sizeof(arena_run_t) + (sizeof(unsigned) *
(try_mask_nelms - 1)) > try_reg0_offset);
} while (try_run_size <= arena_maxclass && try_run_size <= RUN_MAX_SMALL
Use fixed point integer math instead of floating point math when calculating run sizes. Use of the floating point unit was a potential pessimization to context switching for applications that do not otherwise use floating point math. [1] Reformat cpp macro-related comments to improve consistency. Submitted by: das
2007-12-18 05:27:57 +00:00
&& RUN_MAX_OVRHD * (bin->reg_size << 3) > RUN_MAX_OVRHD_RELAX
&& (try_reg0_offset << RUN_BFP) > RUN_MAX_OVRHD * try_run_size);
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
assert(sizeof(arena_run_t) + (sizeof(unsigned) * (good_mask_nelms - 1))
<= good_reg0_offset);
assert((good_mask_nelms << (SIZEOF_INT_2POW + 3)) >= good_nregs);
/* Copy final settings. */
bin->run_size = good_run_size;
bin->nregs = good_nregs;
bin->regs_mask_nelms = good_mask_nelms;
bin->reg0_offset = good_reg0_offset;
return (good_run_size);
}
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
static inline void
arena_lock_balance(arena_t *arena)
{
unsigned contention;
contention = malloc_spin_lock(&arena->lock);
if (narenas > 1) {
/*
* Calculate the exponentially averaged contention for this
* arena. Due to integer math always rounding down, this value
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* decays somewhat faster than normal.
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
*/
arena->contention = (((uint64_t)arena->contention
* (uint64_t)((1U << BALANCE_ALPHA_INV_2POW)-1))
+ (uint64_t)contention) >> BALANCE_ALPHA_INV_2POW;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (arena->contention >= opt_balance_threshold)
arena_lock_balance_hard(arena);
}
}
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static void
arena_lock_balance_hard(arena_t *arena)
{
uint32_t ind;
arena->contention = 0;
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
#ifdef MALLOC_STATS
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena->stats.nbalance++;
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
ind = PRN(balance, narenas_2pow);
if (arenas[ind] != NULL)
arenas_map = arenas[ind];
else {
malloc_spin_lock(&arenas_lock);
if (arenas[ind] != NULL)
arenas_map = arenas[ind];
else
arenas_map = arenas_extend(ind);
malloc_spin_unlock(&arenas_lock);
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
}
}
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#endif
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
static inline void *
mag_alloc(mag_t *mag)
{
if (mag->nrounds == 0)
return (NULL);
mag->nrounds--;
return (mag->rounds[mag->nrounds]);
}
static void
mag_load(mag_t *mag)
{
arena_t *arena;
arena_bin_t *bin;
arena_run_t *run;
void *round;
size_t i;
arena = choose_arena();
bin = &arena->bins[mag->binind];
#ifdef MALLOC_BALANCE
arena_lock_balance(arena);
#else
malloc_spin_lock(&arena->lock);
#endif
for (i = mag->nrounds; i < max_rounds; i++) {
if ((run = bin->runcur) != NULL && run->nfree > 0)
round = arena_bin_malloc_easy(arena, bin, run);
else
round = arena_bin_malloc_hard(arena, bin);
if (round == NULL)
break;
mag->rounds[i] = round;
}
#ifdef MALLOC_STATS
bin->stats.nmags++;
arena->stats.nmalloc_small += (i - mag->nrounds);
arena->stats.allocated_small += (i - mag->nrounds) * bin->reg_size;
#endif
malloc_spin_unlock(&arena->lock);
mag->nrounds = i;
}
static inline void *
mag_rack_alloc(mag_rack_t *rack, size_t size, bool zero)
{
void *ret;
bin_mags_t *bin_mags;
mag_t *mag;
size_t binind;
binind = size2bin[size];
assert(binind < nbins);
bin_mags = &rack->bin_mags[binind];
mag = bin_mags->curmag;
if (mag == NULL) {
/* Create an initial magazine for this size class. */
assert(bin_mags->sparemag == NULL);
mag = mag_create(choose_arena(), binind);
if (mag == NULL)
return (NULL);
bin_mags->curmag = mag;
mag_load(mag);
}
ret = mag_alloc(mag);
if (ret == NULL) {
if (bin_mags->sparemag != NULL) {
if (bin_mags->sparemag->nrounds > 0) {
/* Swap magazines. */
bin_mags->curmag = bin_mags->sparemag;
bin_mags->sparemag = mag;
mag = bin_mags->curmag;
} else {
/* Reload the current magazine. */
mag_load(mag);
}
} else {
/* Create a second magazine. */
mag = mag_create(choose_arena(), binind);
if (mag == NULL)
return (NULL);
mag_load(mag);
bin_mags->sparemag = bin_mags->curmag;
bin_mags->curmag = mag;
}
ret = mag_alloc(mag);
if (ret == NULL)
return (NULL);
}
if (zero == false) {
if (opt_junk)
memset(ret, 0xa5, size);
else if (opt_zero)
memset(ret, 0, size);
} else
memset(ret, 0, size);
return (ret);
}
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static inline void *
arena_malloc_small(arena_t *arena, size_t size, bool zero)
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
{
void *ret;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_bin_t *bin;
arena_run_t *run;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
size_t binind;
Various cosmetics Improve chances of troublefree 64bit operation. [imp] Noticed by: Warner Losh <imp@village.org>
1996-10-26 08:19:07 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
binind = size2bin[size];
assert(binind < nbins);
bin = &arena->bins[binind];
size = bin->reg_size;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_lock_balance(arena);
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#else
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_spin_lock(&arena->lock);
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if ((run = bin->runcur) != NULL && run->nfree > 0)
ret = arena_bin_malloc_easy(arena, bin, run);
else
ret = arena_bin_malloc_hard(arena, bin);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (ret == NULL) {
malloc_spin_unlock(&arena->lock);
return (NULL);
}
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
bin->stats.nrequests++;
arena->stats.nmalloc_small++;
arena->stats.allocated_small += size;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_spin_unlock(&arena->lock);
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (zero == false) {
if (opt_junk)
memset(ret, 0xa5, size);
else if (opt_zero)
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
memset(ret, 0, size);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
} else
memset(ret, 0, size);
return (ret);
}
static void *
arena_malloc_large(arena_t *arena, size_t size, bool zero)
{
void *ret;
/* Large allocation. */
size = PAGE_CEILING(size);
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_lock_balance(arena);
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#else
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_spin_lock(&arena->lock);
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#endif
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
ret = (void *)arena_run_alloc(arena, size, true, zero);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (ret == NULL) {
malloc_spin_unlock(&arena->lock);
return (NULL);
}
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#ifdef MALLOC_STATS
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena->stats.nmalloc_large++;
arena->stats.allocated_large += size;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
malloc_spin_unlock(&arena->lock);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (zero == false) {
if (opt_junk)
memset(ret, 0xa5, size);
else if (opt_zero)
memset(ret, 0, size);
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
return (ret);
}
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
static inline void *
arena_malloc(arena_t *arena, size_t size, bool zero)
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(arena != NULL);
assert(arena->magic == ARENA_MAGIC);
assert(size != 0);
assert(QUANTUM_CEILING(size) <= arena_maxclass);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (size <= bin_maxclass) {
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
if (__isthreaded && opt_mag) {
mag_rack_t *rack = mag_rack;
if (rack == NULL) {
rack = mag_rack_create(arena);
if (rack == NULL)
return (NULL);
mag_rack = rack;
}
return (mag_rack_alloc(rack, size, zero));
} else
#endif
return (arena_malloc_small(arena, size, zero));
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
} else
return (arena_malloc_large(arena, size, zero));
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
static inline void *
imalloc(size_t size)
{
assert(size != 0);
if (size <= arena_maxclass)
return (arena_malloc(choose_arena(), size, false));
else
return (huge_malloc(size, false));
}
static inline void *
icalloc(size_t size)
{
if (size <= arena_maxclass)
return (arena_malloc(choose_arena(), size, true));
else
return (huge_malloc(size, true));
}
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
/* Only handles large allocations that require more than page alignment. */
static void *
arena_palloc(arena_t *arena, size_t alignment, size_t size, size_t alloc_size)
{
void *ret;
size_t offset;
arena_chunk_t *chunk;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert((size & PAGE_MASK) == 0);
assert((alignment & PAGE_MASK) == 0);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
arena_lock_balance(arena);
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#else
malloc_spin_lock(&arena->lock);
#endif
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
ret = (void *)arena_run_alloc(arena, alloc_size, true, false);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (ret == NULL) {
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_unlock(&arena->lock);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
return (NULL);
}
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ret);
offset = (uintptr_t)ret & (alignment - 1);
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert((offset & PAGE_MASK) == 0);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
assert(offset < alloc_size);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if (offset == 0)
arena_run_trim_tail(arena, chunk, ret, alloc_size, size, false);
else {
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
size_t leadsize, trailsize;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
leadsize = alignment - offset;
if (leadsize > 0) {
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_trim_head(arena, chunk, ret, alloc_size,
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
alloc_size - leadsize);
ret = (void *)((uintptr_t)ret + leadsize);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
}
trailsize = alloc_size - leadsize - size;
if (trailsize != 0) {
/* Trim trailing space. */
assert(trailsize < alloc_size);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_trim_tail(arena, chunk, ret, size + trailsize,
size, false);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
}
}
#ifdef MALLOC_STATS
arena->stats.nmalloc_large++;
arena->stats.allocated_large += size;
#endif
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_unlock(&arena->lock);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (opt_junk)
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
memset(ret, 0xa5, size);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
else if (opt_zero)
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
memset(ret, 0, size);
return (ret);
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
static inline void *
ipalloc(size_t alignment, size_t size)
{
void *ret;
size_t ceil_size;
/*
* Round size up to the nearest multiple of alignment.
*
* This done, we can take advantage of the fact that for each small
* size class, every object is aligned at the smallest power of two
* that is non-zero in the base two representation of the size. For
* example:
*
* Size | Base 2 | Minimum alignment
* -----+----------+------------------
* 96 | 1100000 | 32
* 144 | 10100000 | 32
* 192 | 11000000 | 64
*
* Depending on runtime settings, it is possible that arena_malloc()
* will further round up to a power of two, but that never causes
* correctness issues.
*/
ceil_size = (size + (alignment - 1)) & (-alignment);
/*
* (ceil_size < size) protects against the combination of maximal
* alignment and size greater than maximal alignment.
*/
if (ceil_size < size) {
/* size_t overflow. */
return (NULL);
}
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
if (ceil_size <= PAGE_SIZE || (alignment <= PAGE_SIZE
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
&& ceil_size <= arena_maxclass))
ret = arena_malloc(choose_arena(), ceil_size, false);
else {
size_t run_size;
/*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* We can't achieve subpage alignment, so round up alignment
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
* permanently; it makes later calculations simpler.
*/
alignment = PAGE_CEILING(alignment);
ceil_size = PAGE_CEILING(size);
/*
* (ceil_size < size) protects against very large sizes within
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
* PAGE_SIZE of SIZE_T_MAX.
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
*
* (ceil_size + alignment < ceil_size) protects against the
* combination of maximal alignment and ceil_size large enough
* to cause overflow. This is similar to the first overflow
* check above, but it needs to be repeated due to the new
* ceil_size value, which may now be *equal* to maximal
* alignment, whereas before we only detected overflow if the
* original size was *greater* than maximal alignment.
*/
if (ceil_size < size || ceil_size + alignment < ceil_size) {
/* size_t overflow. */
return (NULL);
}
/*
* Calculate the size of the over-size run that arena_palloc()
* would need to allocate in order to guarantee the alignment.
*/
if (ceil_size >= alignment)
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run_size = ceil_size + alignment - PAGE_SIZE;
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
else {
/*
* It is possible that (alignment << 1) will cause
* overflow, but it doesn't matter because we also
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
* subtract PAGE_SIZE, which in the case of overflow
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
* leaves us with a very large run_size. That causes
* the first conditional below to fail, which means
* that the bogus run_size value never gets used for
* anything important.
*/
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run_size = (alignment << 1) - PAGE_SIZE;
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
}
if (run_size <= arena_maxclass) {
ret = arena_palloc(choose_arena(), alignment, ceil_size,
run_size);
} else if (alignment <= chunksize)
ret = huge_malloc(ceil_size, false);
else
ret = huge_palloc(alignment, ceil_size);
}
assert(((uintptr_t)ret & (alignment - 1)) == 0);
return (ret);
}
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Return the size of the allocation pointed to by ptr. */
static size_t
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arena_salloc(const void *ptr)
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
{
size_t ret;
arena_chunk_t *chunk;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
size_t pageind, mapbits;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
assert(ptr != NULL);
assert(CHUNK_ADDR2BASE(ptr) != ptr);
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
pageind = (((uintptr_t)ptr - (uintptr_t)chunk) >> PAGE_SHIFT);
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
mapbits = chunk->map[pageind].bits;
assert((mapbits & CHUNK_MAP_ALLOCATED) != 0);
if ((mapbits & CHUNK_MAP_LARGE) == 0) {
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
arena_run_t *run = (arena_run_t *)(mapbits & ~PAGE_MASK);
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
assert(run->magic == ARENA_RUN_MAGIC);
ret = run->bin->reg_size;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
} else {
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
ret = mapbits & ~PAGE_MASK;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
assert(ret != 0);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
return (ret);
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
static inline size_t
isalloc(const void *ptr)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
{
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
size_t ret;
arena_chunk_t *chunk;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
assert(ptr != NULL);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
if (chunk != ptr) {
/* Region. */
assert(chunk->arena->magic == ARENA_MAGIC);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
ret = arena_salloc(ptr);
} else {
extent_node_t *node, key;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
/* Chunk (huge allocation). */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
malloc_mutex_lock(&huge_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
/* Extract from tree of huge allocations. */
key.addr = __DECONST(void *, ptr);
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
node = extent_tree_ad_search(&huge, &key);
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
assert(node != NULL);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
ret = node->size;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
malloc_mutex_unlock(&huge_mtx);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
return (ret);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
static inline void
arena_dalloc_small(arena_t *arena, arena_chunk_t *chunk, void *ptr,
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_chunk_map_t *mapelm)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
{
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
arena_run_t *run;
arena_bin_t *bin;
size_t size;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run = (arena_run_t *)(mapelm->bits & ~PAGE_MASK);
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
assert(run->magic == ARENA_RUN_MAGIC);
bin = run->bin;
size = bin->reg_size;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
if (opt_junk)
memset(ptr, 0x5a, size);
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
arena_run_reg_dalloc(run, bin, ptr, size);
run->nfree++;
if (run->nfree == bin->nregs) {
/* Deallocate run. */
if (run == bin->runcur)
bin->runcur = NULL;
else if (bin->nregs != 1) {
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
size_t run_pageind = (((uintptr_t)run -
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(uintptr_t)chunk)) >> PAGE_SHIFT;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_chunk_map_t *run_mapelm =
&chunk->map[run_pageind];
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
/*
* This block's conditional is necessary because if the
* run only contains one region, then it never gets
* inserted into the non-full runs tree.
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_tree_remove(&bin->runs, run_mapelm);
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
}
#ifdef MALLOC_DEBUG
run->magic = 0;
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_run_dalloc(arena, run, true);
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
#ifdef MALLOC_STATS
bin->stats.curruns--;
#endif
} else if (run->nfree == 1 && run != bin->runcur) {
/*
* Make sure that bin->runcur always refers to the lowest
* non-full run, if one exists.
*/
if (bin->runcur == NULL)
bin->runcur = run;
else if ((uintptr_t)run < (uintptr_t)bin->runcur) {
/* Switch runcur. */
if (bin->runcur->nfree > 0) {
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_chunk_t *runcur_chunk =
CHUNK_ADDR2BASE(bin->runcur);
size_t runcur_pageind =
(((uintptr_t)bin->runcur -
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(uintptr_t)runcur_chunk)) >> PAGE_SHIFT;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_chunk_map_t *runcur_mapelm =
&runcur_chunk->map[runcur_pageind];
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
/* Insert runcur. */
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_tree_insert(&bin->runs,
runcur_mapelm);
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
}
bin->runcur = run;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
} else {
size_t run_pageind = (((uintptr_t)run -
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(uintptr_t)chunk)) >> PAGE_SHIFT;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_chunk_map_t *run_mapelm =
&chunk->map[run_pageind];
assert(arena_run_tree_search(&bin->runs, run_mapelm) ==
NULL);
arena_run_tree_insert(&bin->runs, run_mapelm);
}
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
}
#ifdef MALLOC_STATS
arena->stats.allocated_small -= size;
arena->stats.ndalloc_small++;
#endif
}
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
static void
mag_unload(mag_t *mag)
{
arena_chunk_t *chunk;
arena_t *arena;
void *round;
size_t i, ndeferred, nrounds;
for (ndeferred = mag->nrounds; ndeferred > 0;) {
nrounds = ndeferred;
/* Lock the arena associated with the first round. */
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(mag->rounds[0]);
arena = chunk->arena;
#ifdef MALLOC_BALANCE
arena_lock_balance(arena);
#else
malloc_spin_lock(&arena->lock);
#endif
/* Deallocate every round that belongs to the locked arena. */
for (i = ndeferred = 0; i < nrounds; i++) {
round = mag->rounds[i];
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(round);
if (chunk->arena == arena) {
size_t pageind = (((uintptr_t)round -
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
(uintptr_t)chunk) >> PAGE_SHIFT);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
arena_chunk_map_t *mapelm =
&chunk->map[pageind];
arena_dalloc_small(arena, chunk, round, mapelm);
} else {
/*
* This round was allocated via a different
* arena than the one that is currently locked.
* Stash the round, so that it can be handled
* in a future pass.
*/
mag->rounds[ndeferred] = round;
ndeferred++;
}
}
malloc_spin_unlock(&arena->lock);
}
mag->nrounds = 0;
}
static inline void
mag_rack_dalloc(mag_rack_t *rack, void *ptr)
{
arena_t *arena;
arena_chunk_t *chunk;
arena_run_t *run;
arena_bin_t *bin;
bin_mags_t *bin_mags;
mag_t *mag;
size_t pageind, binind;
arena_chunk_map_t *mapelm;
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
arena = chunk->arena;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
pageind = (((uintptr_t)ptr - (uintptr_t)chunk) >> PAGE_SHIFT);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
mapelm = &chunk->map[pageind];
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
run = (arena_run_t *)(mapelm->bits & ~PAGE_MASK);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
assert(run->magic == ARENA_RUN_MAGIC);
bin = run->bin;
binind = ((uintptr_t)bin - (uintptr_t)&arena->bins) /
sizeof(arena_bin_t);
assert(binind < nbins);
if (opt_junk)
memset(ptr, 0x5a, arena->bins[binind].reg_size);
bin_mags = &rack->bin_mags[binind];
mag = bin_mags->curmag;
if (mag == NULL) {
/* Create an initial magazine for this size class. */
assert(bin_mags->sparemag == NULL);
mag = mag_create(choose_arena(), binind);
if (mag == NULL) {
malloc_spin_lock(&arena->lock);
arena_dalloc_small(arena, chunk, ptr, mapelm);
malloc_spin_unlock(&arena->lock);
return;
}
bin_mags->curmag = mag;
}
if (mag->nrounds == max_rounds) {
if (bin_mags->sparemag != NULL) {
if (bin_mags->sparemag->nrounds < max_rounds) {
/* Swap magazines. */
bin_mags->curmag = bin_mags->sparemag;
bin_mags->sparemag = mag;
mag = bin_mags->curmag;
} else {
/* Unload the current magazine. */
mag_unload(mag);
}
} else {
/* Create a second magazine. */
mag = mag_create(choose_arena(), binind);
if (mag == NULL) {
mag = rack->bin_mags[binind].curmag;
mag_unload(mag);
} else {
bin_mags->sparemag = bin_mags->curmag;
bin_mags->curmag = mag;
}
}
assert(mag->nrounds < max_rounds);
}
mag->rounds[mag->nrounds] = ptr;
mag->nrounds++;
}
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
static void
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_dalloc_large(arena_t *arena, arena_chunk_t *chunk, void *ptr)
{
/* Large allocation. */
malloc_spin_lock(&arena->lock);
#ifndef MALLOC_STATS
if (opt_junk)
#endif
{
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
size_t pageind = ((uintptr_t)ptr - (uintptr_t)chunk) >>
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_SHIFT;
size_t size = chunk->map[pageind].bits & ~PAGE_MASK;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
#ifdef MALLOC_STATS
if (opt_junk)
#endif
memset(ptr, 0x5a, size);
#ifdef MALLOC_STATS
arena->stats.allocated_large -= size;
#endif
}
#ifdef MALLOC_STATS
arena->stats.ndalloc_large++;
#endif
arena_run_dalloc(arena, (arena_run_t *)ptr, true);
malloc_spin_unlock(&arena->lock);
}
static inline void
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
arena_dalloc(arena_t *arena, arena_chunk_t *chunk, void *ptr)
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
size_t pageind;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena_chunk_map_t *mapelm;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(arena != NULL);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
assert(arena->magic == ARENA_MAGIC);
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
assert(chunk->arena == arena);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(ptr != NULL);
assert(CHUNK_ADDR2BASE(ptr) != ptr);
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
pageind = (((uintptr_t)ptr - (uintptr_t)chunk) >> PAGE_SHIFT);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
mapelm = &chunk->map[pageind];
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
assert((mapelm->bits & CHUNK_MAP_ALLOCATED) != 0);
if ((mapelm->bits & CHUNK_MAP_LARGE) == 0) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Small allocation. */
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
if (__isthreaded && opt_mag) {
mag_rack_t *rack = mag_rack;
if (rack == NULL) {
rack = mag_rack_create(arena);
if (rack == NULL) {
malloc_spin_lock(&arena->lock);
arena_dalloc_small(arena, chunk, ptr,
mapelm);
malloc_spin_unlock(&arena->lock);
}
mag_rack = rack;
}
mag_rack_dalloc(rack, ptr);
} else {
#endif
malloc_spin_lock(&arena->lock);
arena_dalloc_small(arena, chunk, ptr, mapelm);
malloc_spin_unlock(&arena->lock);
#ifdef MALLOC_MAG
}
#endif
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
} else
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena_dalloc_large(arena, chunk, ptr);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
static inline void
idalloc(void *ptr)
{
arena_chunk_t *chunk;
assert(ptr != NULL);
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
if (chunk != ptr)
arena_dalloc(chunk->arena, chunk, ptr);
else
huge_dalloc(ptr);
}
static void
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
arena_ralloc_large_shrink(arena_t *arena, arena_chunk_t *chunk, void *ptr,
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
size_t size, size_t oldsize)
{
assert(size < oldsize);
/*
* Shrink the run, and make trailing pages available for other
* allocations.
*/
#ifdef MALLOC_BALANCE
arena_lock_balance(arena);
#else
malloc_spin_lock(&arena->lock);
#endif
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_trim_tail(arena, chunk, (arena_run_t *)ptr, oldsize, size,
true);
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
#ifdef MALLOC_STATS
arena->stats.allocated_large -= oldsize - size;
#endif
malloc_spin_unlock(&arena->lock);
}
static bool
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
arena_ralloc_large_grow(arena_t *arena, arena_chunk_t *chunk, void *ptr,
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
size_t size, size_t oldsize)
{
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
size_t pageind = ((uintptr_t)ptr - (uintptr_t)chunk) >> PAGE_SHIFT;
size_t npages = oldsize >> PAGE_SHIFT;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert(oldsize == (chunk->map[pageind].bits & ~PAGE_MASK));
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
/* Try to extend the run. */
assert(size > oldsize);
#ifdef MALLOC_BALANCE
arena_lock_balance(arena);
#else
malloc_spin_lock(&arena->lock);
#endif
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
if (pageind + npages < chunk_npages && (chunk->map[pageind+npages].bits
& CHUNK_MAP_ALLOCATED) == 0 && (chunk->map[pageind+npages].bits &
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
~PAGE_MASK) >= size - oldsize) {
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
/*
* The next run is available and sufficiently large. Split the
* following run, then merge the first part with the existing
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
* allocation.
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
*/
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_run_split(arena, (arena_run_t *)((uintptr_t)chunk +
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
((pageind+npages) << PAGE_SHIFT)), size - oldsize, true,
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
false);
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
chunk->map[pageind].bits = size | CHUNK_MAP_LARGE |
CHUNK_MAP_ALLOCATED;
chunk->map[pageind+npages].bits = CHUNK_MAP_LARGE |
CHUNK_MAP_ALLOCATED;
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
#ifdef MALLOC_STATS
arena->stats.allocated_large += size - oldsize;
#endif
malloc_spin_unlock(&arena->lock);
return (false);
}
malloc_spin_unlock(&arena->lock);
return (true);
}
/*
* Try to resize a large allocation, in order to avoid copying. This will
* always fail if growing an object, and the following run is already in use.
*/
static bool
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
arena_ralloc_large(void *ptr, size_t size, size_t oldsize)
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
{
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
size_t psize;
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
psize = PAGE_CEILING(size);
if (psize == oldsize) {
/* Same size class. */
if (opt_junk && size < oldsize) {
memset((void *)((uintptr_t)ptr + size), 0x5a, oldsize -
size);
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
return (false);
} else {
Fix a race condition in arena_ralloc() for shrinking in-place large reallocation, when junk filling is enabled. Junk filling must occur prior to shrinking, since any deallocated trailing pages are immediately available for use by other threads. Reported by: Mats Palmgren <mats.palmgren@bredband.net>
2008-02-17 18:34:17 +00:00
arena_chunk_t *chunk;
arena_t *arena;
chunk = (arena_chunk_t *)CHUNK_ADDR2BASE(ptr);
arena = chunk->arena;
assert(arena->magic == ARENA_MAGIC);
if (psize < oldsize) {
/* Fill before shrinking in order avoid a race. */
if (opt_junk) {
memset((void *)((uintptr_t)ptr + size), 0x5a,
oldsize - size);
}
arena_ralloc_large_shrink(arena, chunk, ptr, psize,
oldsize);
return (false);
} else {
bool ret = arena_ralloc_large_grow(arena, chunk, ptr,
psize, oldsize);
if (ret == false && opt_zero) {
memset((void *)((uintptr_t)ptr + oldsize), 0,
size - oldsize);
}
return (ret);
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
}
}
static void *
arena_ralloc(void *ptr, size_t size, size_t oldsize)
{
void *ret;
size_t copysize;
/* Try to avoid moving the allocation. */
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
if (size <= bin_maxclass) {
if (oldsize <= bin_maxclass && size2bin[size] ==
size2bin[oldsize])
goto IN_PLACE;
} else {
if (oldsize > bin_maxclass && oldsize <= arena_maxclass) {
assert(size > bin_maxclass);
if (arena_ralloc_large(ptr, size, oldsize) == false)
return (ptr);
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
}
/*
* If we get here, then size and oldsize are different enough that we
* need to move the object. In that case, fall back to allocating new
* space and copying.
*/
ret = arena_malloc(choose_arena(), size, false);
if (ret == NULL)
return (NULL);
/* Junk/zero-filling were already done by arena_malloc(). */
copysize = (size < oldsize) ? size : oldsize;
memcpy(ret, ptr, copysize);
idalloc(ptr);
return (ret);
IN_PLACE:
if (opt_junk && size < oldsize)
memset((void *)((uintptr_t)ptr + size), 0x5a, oldsize - size);
else if (opt_zero && size > oldsize)
memset((void *)((uintptr_t)ptr + oldsize), 0, size - oldsize);
return (ptr);
}
static inline void *
iralloc(void *ptr, size_t size)
{
size_t oldsize;
assert(ptr != NULL);
assert(size != 0);
oldsize = isalloc(ptr);
if (size <= arena_maxclass)
return (arena_ralloc(ptr, size, oldsize));
else
return (huge_ralloc(ptr, size, oldsize));
}
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
static bool
arena_new(arena_t *arena)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
unsigned i;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
arena_bin_t *bin;
Remove variables which are assigned values and never used thereafter. Found by: LLVM/Clang Static Checker Approved by: jasone
2008-08-08 20:42:42 +00:00
size_t prev_run_size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
if (malloc_spin_init(&arena->lock))
return (true);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
memset(&arena->stats, 0, sizeof(arena_stats_t));
#endif
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Initialize chunks. */
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
arena_chunk_tree_dirty_new(&arena->chunks_dirty);
Implement chunk allocation/deallocation hysteresis by caching one spare chunk per arena, rather than immediately deallocating all unused chunks. This fixes a potential performance issue when allocating/deallocating an object of size (4kB..1MB] in a loop. Reported by: davidxu
2006-12-23 00:18:51 +00:00
arena->spare = NULL;
Malloc flag X makes malloc behave like the canonical xmalloc() wrapper. Untested support for Solaris from John-Mark Gurney Reviewed by: phk Submitted by: (partially) John-Mark Gurney <gurney_j@resnet.uoregon.edu>
1997-05-30 20:39:32 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
arena->ndirty = 0;
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
arena_avail_tree_new(&arena->runs_avail);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
arena->contention = 0;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#endif
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Initialize bins. */
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
prev_run_size = PAGE_SIZE;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
i = 0;
#ifdef MALLOC_TINY
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* (2^n)-spaced tiny bins. */
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
for (; i < ntbins; i++) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
bin = &arena->bins[i];
bin->runcur = NULL;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
arena_run_tree_new(&bin->runs);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
bin->reg_size = (1U << (TINY_MIN_2POW + i));
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
prev_run_size = arena_bin_run_size_calc(bin, prev_run_size);
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
memset(&bin->stats, 0, sizeof(malloc_bin_stats_t));
#endif
}
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Quantum-spaced bins. */
for (; i < ntbins + nqbins; i++) {
bin = &arena->bins[i];
bin->runcur = NULL;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
arena_run_tree_new(&bin->runs);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
bin->reg_size = (i - ntbins + 1) << QUANTUM_2POW;
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
prev_run_size = arena_bin_run_size_calc(bin, prev_run_size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
memset(&bin->stats, 0, sizeof(malloc_bin_stats_t));
#endif
}
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
/* Cacheline-spaced bins. */
for (; i < ntbins + nqbins + ncbins; i++) {
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
bin = &arena->bins[i];
bin->runcur = NULL;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
arena_run_tree_new(&bin->runs);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
bin->reg_size = cspace_min + ((i - (ntbins + nqbins)) <<
CACHELINE_2POW);
prev_run_size = arena_bin_run_size_calc(bin, prev_run_size);
#ifdef MALLOC_STATS
memset(&bin->stats, 0, sizeof(malloc_bin_stats_t));
#endif
}
/* Subpage-spaced bins. */
for (; i < nbins; i++) {
bin = &arena->bins[i];
bin->runcur = NULL;
arena_run_tree_new(&bin->runs);
bin->reg_size = sspace_min + ((i - (ntbins + nqbins + ncbins))
<< SUBPAGE_2POW);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
prev_run_size = arena_bin_run_size_calc(bin, prev_run_size);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
#ifdef MALLOC_STATS
memset(&bin->stats, 0, sizeof(malloc_bin_stats_t));
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_DEBUG
arena->magic = ARENA_MAGIC;
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
return (false);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Create a new arena and insert it into the arenas array at index ind. */
static arena_t *
arenas_extend(unsigned ind)
{
arena_t *ret;
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Allocate enough space for trailing bins. */
ret = (arena_t *)base_alloc(sizeof(arena_t)
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
+ (sizeof(arena_bin_t) * (nbins - 1)));
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (ret != NULL && arena_new(ret) == false) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
arenas[ind] = ret;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
return (ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/* Only reached if there is an OOM error. */
Preallocate a small structure, so we can sbrk(2) further back. Reviewed by: phk Submitted by: Kaleb Keithly <kaleb@x.org>
1995-12-18 12:03:54 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/*
* OOM here is quite inconvenient to propagate, since dealing with it
* would require a check for failure in the fast path. Instead, punt
* by using arenas[0]. In practice, this is an extremely unlikely
* failure.
*/
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error initializing arena\n", "", "");
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
if (opt_abort)
abort();
return (arenas[0]);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
static mag_t *
mag_create(arena_t *arena, size_t binind)
{
mag_t *ret;
if (sizeof(mag_t) + (sizeof(void *) * (max_rounds - 1)) <=
bin_maxclass) {
ret = arena_malloc_small(arena, sizeof(mag_t) + (sizeof(void *)
* (max_rounds - 1)), false);
} else {
ret = imalloc(sizeof(mag_t) + (sizeof(void *) * (max_rounds -
1)));
}
if (ret == NULL)
return (NULL);
ret->binind = binind;
ret->nrounds = 0;
return (ret);
}
static void
mag_destroy(mag_t *mag)
{
arena_t *arena;
arena_chunk_t *chunk;
size_t pageind;
arena_chunk_map_t *mapelm;
chunk = CHUNK_ADDR2BASE(mag);
arena = chunk->arena;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
pageind = (((uintptr_t)mag - (uintptr_t)chunk) >> PAGE_SHIFT);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
mapelm = &chunk->map[pageind];
assert(mag->nrounds == 0);
if (sizeof(mag_t) + (sizeof(void *) * (max_rounds - 1)) <=
bin_maxclass) {
malloc_spin_lock(&arena->lock);
arena_dalloc_small(arena, chunk, mag, mapelm);
malloc_spin_unlock(&arena->lock);
} else
idalloc(mag);
}
static mag_rack_t *
mag_rack_create(arena_t *arena)
{
assert(sizeof(mag_rack_t) + (sizeof(bin_mags_t *) * (nbins - 1)) <=
bin_maxclass);
return (arena_malloc_small(arena, sizeof(mag_rack_t) +
(sizeof(bin_mags_t) * (nbins - 1)), true));
}
static void
mag_rack_destroy(mag_rack_t *rack)
{
arena_t *arena;
arena_chunk_t *chunk;
bin_mags_t *bin_mags;
size_t i, pageind;
arena_chunk_map_t *mapelm;
for (i = 0; i < nbins; i++) {
bin_mags = &rack->bin_mags[i];
if (bin_mags->curmag != NULL) {
assert(bin_mags->curmag->binind == i);
mag_unload(bin_mags->curmag);
mag_destroy(bin_mags->curmag);
}
if (bin_mags->sparemag != NULL) {
assert(bin_mags->sparemag->binind == i);
mag_unload(bin_mags->sparemag);
mag_destroy(bin_mags->sparemag);
}
}
chunk = CHUNK_ADDR2BASE(rack);
arena = chunk->arena;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
pageind = (((uintptr_t)rack - (uintptr_t)chunk) >> PAGE_SHIFT);
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
mapelm = &chunk->map[pageind];
malloc_spin_lock(&arena->lock);
arena_dalloc_small(arena, chunk, rack, mapelm);
malloc_spin_unlock(&arena->lock);
}
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* End arena.
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
*/
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/******************************************************************************/
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* Begin general internal functions.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
static void *
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
huge_malloc(size_t size, bool zero)
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
size_t csize;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_node_t *node;
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
/* Allocate one or more contiguous chunks for this request. */
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
csize = CHUNK_CEILING(size);
if (csize == 0) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* size is large enough to cause size_t wrap-around. */
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
return (NULL);
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Allocate an extent node with which to track the chunk. */
node = base_node_alloc();
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
if (node == NULL)
return (NULL);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
ret = chunk_alloc(csize, zero);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (ret == NULL) {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_node_dealloc(node);
Remove unwarranted uses of 'goto'.
2006-01-27 07:46:22 +00:00
return (NULL);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
}
Add a missing case for the switch statement in arena_run_reg_dalloc(). [1] Fix a leak in chunk_dealloc(). [2] Reported by: [1] djam8193ah@hotmail.com, [2] Ville-Pertti Keinonen <will@exomi.com>
2006-06-20 20:38:25 +00:00
/* Insert node into huge. */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
node->addr = ret;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
node->size = csize;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_lock(&huge_mtx);
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_ad_insert(&huge, node);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
huge_nmalloc++;
Change the semantics of brk_max to dynamically deal with data segment bounds. [1] Modify logic for utilizing the data segment, such that it is possible to create huge allocations there. Shrink the data segment when deallocating a chunk, if it is at the end of the data segment. Rename chunk_size to csize in huge_malloc(), in order to avoid masking a static variable of the same name. [1] Reported by: Paul Allen <nospam@ugcs.caltech.edu>
2006-04-27 01:03:00 +00:00
huge_allocated += csize;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_unlock(&huge_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
if (zero == false) {
if (opt_junk)
memset(ret, 0xa5, csize);
else if (opt_zero)
memset(ret, 0, csize);
}
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
return (ret);
}
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
/* Only handles large allocations that require more than chunk alignment. */
static void *
huge_palloc(size_t alignment, size_t size)
{
void *ret;
size_t alloc_size, chunk_size, offset;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_node_t *node;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
/*
* This allocation requires alignment that is even larger than chunk
* alignment. This means that huge_malloc() isn't good enough.
*
* Allocate almost twice as many chunks as are demanded by the size or
* alignment, in order to assure the alignment can be achieved, then
* unmap leading and trailing chunks.
*/
Fix some subtle bugs for posix_memalign() having to do with integer rounding and overflow. Carefully document what the various overflow tests actually detect. The bugs mostly canceled out, such that the worst possible failure cases resulted in non-fatal over-allocations.
2007-03-24 20:44:06 +00:00
assert(alignment >= chunksize);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunk_size = CHUNK_CEILING(size);
if (size >= alignment)
alloc_size = chunk_size + alignment - chunksize;
else
alloc_size = (alignment << 1) - chunksize;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/* Allocate an extent node with which to track the chunk. */
node = base_node_alloc();
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (node == NULL)
return (NULL);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
ret = chunk_alloc(alloc_size, false);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (ret == NULL) {
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_node_dealloc(node);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
return (NULL);
}
offset = (uintptr_t)ret & (alignment - 1);
assert((offset & chunksize_mask) == 0);
assert(offset < alloc_size);
if (offset == 0) {
/* Trim trailing space. */
chunk_dealloc((void *)((uintptr_t)ret + chunk_size), alloc_size
- chunk_size);
} else {
size_t trailsize;
/* Trim leading space. */
chunk_dealloc(ret, alignment - offset);
ret = (void *)((uintptr_t)ret + (alignment - offset));
trailsize = alloc_size - (alignment - offset) - chunk_size;
if (trailsize != 0) {
/* Trim trailing space. */
assert(trailsize < alloc_size);
chunk_dealloc((void *)((uintptr_t)ret + chunk_size),
trailsize);
}
}
/* Insert node into huge. */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
node->addr = ret;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
node->size = chunk_size;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_lock(&huge_mtx);
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_ad_insert(&huge, node);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
#ifdef MALLOC_STATS
Remove the run promotion/demotion machinery. Replace it with red-black trees that track all non-full runs for each bin. Use the red-black trees to be able to guarantee that each new allocation is placed in the lowest address available in any non-full run. This change completes the transition to allocating from low addresses in order to reduce the retention of sparsely used chunks. If the run in current use by a bin becomes empty, deallocate the run rather than retaining it for later use. The previous behavior had the tendency to spread empty runs across multiple chunks, thus preventing the release of chunks that were completely unused. Generalize base_chunk_alloc() (and rename it to base_pages_alloc()) to handle allocation sizes larger than the chunk size, so that it is possible to support chunk sizes that are smaller than an arena object. Reduce the minimum chunk size from 64kB to 8kB. Optimize tracking of addresses for deleted chunks. Fix a statistics bug for huge allocations.
2007-03-28 19:55:07 +00:00
huge_nmalloc++;
huge_allocated += chunk_size;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
#endif
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_unlock(&huge_mtx);
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (opt_junk)
memset(ret, 0xa5, chunk_size);
else if (opt_zero)
memset(ret, 0, chunk_size);
return (ret);
}
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
static void *
huge_ralloc(void *ptr, size_t size, size_t oldsize)
{
void *ret;
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
size_t copysize;
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* Avoid moving the allocation if the size class would not change. */
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
if (oldsize > arena_maxclass &&
Fix junk/zero filling for realloc(). Junk filling was missing in one case, and zero filling was broken in a way that could cause memory corruption. Update comments.
2007-06-15 22:00:16 +00:00
CHUNK_CEILING(size) == CHUNK_CEILING(oldsize)) {
if (opt_junk && size < oldsize) {
memset((void *)((uintptr_t)ptr + size), 0x5a, oldsize
- size);
} else if (opt_zero && size > oldsize) {
memset((void *)((uintptr_t)ptr + oldsize), 0, size
- oldsize);
}
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
return (ptr);
Fix junk/zero filling for realloc(). Junk filling was missing in one case, and zero filling was broken in a way that could cause memory corruption. Update comments.
2007-06-15 22:00:16 +00:00
}
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
/*
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
* If we get here, then size and oldsize are different enough that we
* need to use a different size class. In that case, fall back to
* allocating new space and copying.
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
*/
Avoid re-zeroing memory in calloc() when possible.
2007-11-27 03:12:15 +00:00
ret = huge_malloc(size, false);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
if (ret == NULL)
return (NULL);
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
copysize = (size < oldsize) ? size : oldsize;
memcpy(ret, ptr, copysize);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
idalloc(ptr);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
}
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void
huge_dalloc(void *ptr)
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
extent_node_t *node, key;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_lock(&huge_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Extract from tree of huge allocations. */
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
key.addr = ptr;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
node = extent_tree_ad_search(&huge, &key);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(node != NULL);
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
assert(node->addr == ptr);
Implement red-black trees without using parent pointers, and store the color bit in the least significant bit of the right child pointer, in order to reduce red-black tree linkage overhead by ~2X as compared to sys/tree.h. Use the new red-black tree implementation in malloc, which drops memory usage by ~0.5 or ~1%, for 32- and 64-bit systems, respectively.
2008-04-23 16:09:18 +00:00
extent_tree_ad_remove(&huge, node);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
huge_ndalloc++;
huge_allocated -= node->size;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_unlock(&huge_mtx);
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Unmap chunk. */
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
if (opt_dss && opt_junk)
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
memset(node->addr, 0x5a, node->size);
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
chunk_dealloc(node->addr, node->size);
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_node_dealloc(node);
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
static void
malloc_print_stats(void)
{
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (opt_print_stats) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
char s[UMAX2S_BUFSIZE];
_malloc_message("___ Begin malloc statistics ___\n", "", "",
"");
_malloc_message("Assertions ",
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef NDEBUG
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
"disabled",
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#else
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
"enabled",
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
"\n", "");
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
_malloc_message("Boolean MALLOC_OPTIONS: ",
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
opt_abort ? "A" : "a", "", "");
#ifdef MALLOC_DSS
_malloc_message(opt_dss ? "D" : "d", "", "", "");
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#endif
#ifdef MALLOC_MAG
_malloc_message(opt_mag ? "G" : "g", "", "", "");
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
_malloc_message(opt_junk ? "J" : "j", "", "", "");
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
_malloc_message(opt_mmap ? "M" : "m", "", "", "");
#endif
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
_malloc_message(opt_utrace ? "PU" : "Pu",
opt_sysv ? "V" : "v",
opt_xmalloc ? "X" : "x",
opt_zero ? "Z\n" : "z\n");
_malloc_message("CPUs: ", umax2s(ncpus, s), "\n", "");
_malloc_message("Max arenas: ", umax2s(narenas, s), "\n", "");
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
_malloc_message("Arena balance threshold: ",
umax2s(opt_balance_threshold, s), "\n", "");
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
#endif
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
_malloc_message("Pointer size: ", umax2s(sizeof(void *), s),
"\n", "");
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
_malloc_message("Quantum size: ", umax2s(QUANTUM, s), "\n", "");
_malloc_message("Cacheline size (assumed): ", umax2s(CACHELINE,
s), "\n", "");
#ifdef MALLOC_TINY
_malloc_message("Tiny 2^n-spaced sizes: [", umax2s((1U <<
TINY_MIN_2POW), s), "..", "");
_malloc_message(umax2s((qspace_min >> 1), s), "]\n", "", "");
#endif
_malloc_message("Quantum-spaced sizes: [", umax2s(qspace_min,
s), "..", "");
_malloc_message(umax2s(qspace_max, s), "]\n", "", "");
_malloc_message("Cacheline-spaced sizes: [", umax2s(cspace_min,
s), "..", "");
_malloc_message(umax2s(cspace_max, s), "]\n", "", "");
_malloc_message("Subpage-spaced sizes: [", umax2s(sspace_min,
s), "..", "");
_malloc_message(umax2s(sspace_max, s), "]\n", "", "");
#ifdef MALLOC_MAG
_malloc_message("Rounds per magazine: ", umax2s(max_rounds, s),
"\n", "");
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
_malloc_message("Max dirty pages per arena: ",
umax2s(opt_dirty_max, s), "\n", "");
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
_malloc_message("Chunk size: ", umax2s(chunksize, s), "", "");
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
_malloc_message(" (2^", umax2s(opt_chunk_2pow, s), ")\n", "");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
{
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
size_t allocated, mapped;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
uint64_t nbalance = 0;
#endif
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
unsigned i;
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
arena_t *arena;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* Calculate and print allocated/mapped stats. */
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/* arenas. */
for (i = 0, allocated = 0; i < narenas; i++) {
if (arenas[i] != NULL) {
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_lock(&arenas[i]->lock);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
allocated +=
arenas[i]->stats.allocated_small;
allocated +=
arenas[i]->stats.allocated_large;
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
nbalance += arenas[i]->stats.nbalance;
#endif
malloc_spin_unlock(&arenas[i]->lock);
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
}
}
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
/* huge/base. */
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_lock(&huge_mtx);
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
allocated += huge_allocated;
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
mapped = stats_chunks.curchunks * chunksize;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_unlock(&huge_mtx);
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_mutex_lock(&base_mtx);
mapped += base_mapped;
malloc_mutex_unlock(&base_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_printf("Allocated: %zu, mapped: %zu\n",
allocated, mapped);
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_printf("Arena balance reassignments: %llu\n",
nbalance);
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Print chunk stats. */
{
chunk_stats_t chunks_stats;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_lock(&huge_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
chunks_stats = stats_chunks;
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_unlock(&huge_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_printf("chunks: nchunks "
"highchunks curchunks\n");
malloc_printf(" %13llu%13lu%13lu\n",
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
chunks_stats.nchunks,
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
chunks_stats.highchunks,
chunks_stats.curchunks);
}
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
/* Print chunk stats. */
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
malloc_printf(
"huge: nmalloc ndalloc allocated\n");
malloc_printf(" %12llu %12llu %12zu\n",
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
huge_nmalloc, huge_ndalloc, huge_allocated);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Print stats for each arena. */
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
for (i = 0; i < narenas; i++) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
arena = arenas[i];
if (arena != NULL) {
malloc_printf(
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
"\narenas[%u]:\n", i);
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_lock(&arena->lock);
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
stats_print(arena);
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_unlock(&arena->lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
}
#endif /* #ifdef MALLOC_STATS */
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message("--- End malloc statistics ---\n", "", "", "");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
}
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_DEBUG
static void
size2bin_validate(void)
{
size_t i, size, binind;
assert(size2bin[0] == 0xffU);
i = 1;
# ifdef MALLOC_TINY
/* Tiny. */
for (; i < (1U << TINY_MIN_2POW); i++) {
size = pow2_ceil(1U << TINY_MIN_2POW);
binind = ffs((int)(size >> (TINY_MIN_2POW + 1)));
assert(size2bin[i] == binind);
}
for (; i < qspace_min; i++) {
size = pow2_ceil(i);
binind = ffs((int)(size >> (TINY_MIN_2POW + 1)));
assert(size2bin[i] == binind);
}
# endif
/* Quantum-spaced. */
for (; i <= qspace_max; i++) {
size = QUANTUM_CEILING(i);
binind = ntbins + (size >> QUANTUM_2POW) - 1;
assert(size2bin[i] == binind);
}
/* Cacheline-spaced. */
for (; i <= cspace_max; i++) {
size = CACHELINE_CEILING(i);
binind = ntbins + nqbins + ((size - cspace_min) >>
CACHELINE_2POW);
assert(size2bin[i] == binind);
}
/* Sub-page. */
for (; i <= sspace_max; i++) {
size = SUBPAGE_CEILING(i);
binind = ntbins + nqbins + ncbins + ((size - sspace_min)
>> SUBPAGE_2POW);
assert(size2bin[i] == binind);
}
}
#endif
static bool
size2bin_init(void)
{
if (opt_qspace_max_2pow != QSPACE_MAX_2POW_DEFAULT
|| opt_cspace_max_2pow != CSPACE_MAX_2POW_DEFAULT)
return (size2bin_init_hard());
size2bin = const_size2bin;
#ifdef MALLOC_DEBUG
assert(sizeof(const_size2bin) == bin_maxclass + 1);
size2bin_validate();
#endif
return (false);
}
static bool
size2bin_init_hard(void)
{
size_t i, size, binind;
uint8_t *custom_size2bin;
assert(opt_qspace_max_2pow != QSPACE_MAX_2POW_DEFAULT
|| opt_cspace_max_2pow != CSPACE_MAX_2POW_DEFAULT);
custom_size2bin = (uint8_t *)base_alloc(bin_maxclass + 1);
if (custom_size2bin == NULL)
return (true);
custom_size2bin[0] = 0xffU;
i = 1;
#ifdef MALLOC_TINY
/* Tiny. */
for (; i < (1U << TINY_MIN_2POW); i++) {
size = pow2_ceil(1U << TINY_MIN_2POW);
binind = ffs((int)(size >> (TINY_MIN_2POW + 1)));
custom_size2bin[i] = binind;
}
for (; i < qspace_min; i++) {
size = pow2_ceil(i);
binind = ffs((int)(size >> (TINY_MIN_2POW + 1)));
custom_size2bin[i] = binind;
}
#endif
/* Quantum-spaced. */
for (; i <= qspace_max; i++) {
size = QUANTUM_CEILING(i);
binind = ntbins + (size >> QUANTUM_2POW) - 1;
custom_size2bin[i] = binind;
}
/* Cacheline-spaced. */
for (; i <= cspace_max; i++) {
size = CACHELINE_CEILING(i);
binind = ntbins + nqbins + ((size - cspace_min) >>
CACHELINE_2POW);
custom_size2bin[i] = binind;
}
/* Sub-page. */
for (; i <= sspace_max; i++) {
size = SUBPAGE_CEILING(i);
binind = ntbins + nqbins + ncbins + ((size - sspace_min) >>
SUBPAGE_2POW);
custom_size2bin[i] = binind;
}
size2bin = custom_size2bin;
#ifdef MALLOC_DEBUG
size2bin_validate();
#endif
return (false);
}
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
/*
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
* FreeBSD's pthreads implementation calls malloc(3), so the malloc
* implementation has to take pains to avoid infinite recursion during
* initialization.
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
*/
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
static inline bool
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_init(void)
BSD 4.4 Lite Lib Sources
1994-05-27 05:00:24 +00:00
{
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (malloc_initialized == false)
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (malloc_init_hard());
return (false);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
malloc_pages should be static. Add progname to warning/error message layout. (joerg) Remove inline assembler, no speed impact, not need for the obfuscation (bde) Remove on the fly calculation of parameters, no longer critical. Make D & U flags valid even if we don't support them. Don't call imalloc until we're done initializing. Zap contents on free() if we have "Junk" set. [*] Various nitpicking. [*] As a sideeffect of this change, if you are worried about sensitive data lingering in memory, you can use the 'Junk' option now to make sure phkmalloc zaps memory when it is returned. add char * malloc_options = "J"; to your source. Obviously there is a performance impact.
1996-10-20 13:20:57 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
static bool
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_init_hard(void)
{
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
unsigned i;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
int linklen;
char buf[PATH_MAX + 1];
const char *opts;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
malloc_mutex_lock(&init_lock);
if (malloc_initialized) {
/*
* Another thread initialized the allocator before this one
* acquired init_lock.
*/
malloc_mutex_unlock(&init_lock);
return (false);
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Get number of CPUs. */
{
int mib[2];
size_t len;
mib[0] = CTL_HW;
mib[1] = HW_NCPU;
len = sizeof(ncpus);
if (sysctl(mib, 2, &ncpus, &len, (void *) 0, 0) == -1) {
/* Error. */
ncpus = 1;
}
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
for (i = 0; i < 3; i++) {
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
unsigned j;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Get runtime configuration. */
switch (i) {
case 0:
if ((linklen = readlink("/etc/malloc.conf", buf,
sizeof(buf) - 1)) != -1) {
/*
* Use the contents of the "/etc/malloc.conf"
* symbolic link's name.
*/
buf[linklen] = '\0';
opts = buf;
} else {
/* No configuration specified. */
buf[0] = '\0';
opts = buf;
}
break;
case 1:
if (issetugid() == 0 && (opts =
getenv("MALLOC_OPTIONS")) != NULL) {
/*
* Do nothing; opts is already initialized to
* the value of the MALLOC_OPTIONS environment
* variable.
*/
} else {
/* No configuration specified. */
buf[0] = '\0';
opts = buf;
}
break;
case 2:
if (_malloc_options != NULL) {
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
/*
* Use options that were compiled into the
* program.
*/
opts = _malloc_options;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
} else {
/* No configuration specified. */
buf[0] = '\0';
opts = buf;
}
break;
default:
/* NOTREACHED */
assert(false);
}
for (j = 0; opts[j] != '\0'; j++) {
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
unsigned k, nreps;
bool nseen;
/* Parse repetition count, if any. */
for (nreps = 0, nseen = false;; j++, nseen = true) {
switch (opts[j]) {
case '0': case '1': case '2': case '3':
case '4': case '5': case '6': case '7':
case '8': case '9':
nreps *= 10;
nreps += opts[j] - '0';
break;
default:
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
goto MALLOC_OUT;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
}
}
Clean up manipulation of chunk page map elements to remove some tenuous assumptions about whether bits are set at various times. This makes adding other flags safe. Reorganize functions in order to inline i{m,c,p,s,re}alloc(). This allows the entire fast-path call chains for malloc() and free() to be inlined. [1] Suggested by: [1] Stuart Parmenter <stuart@mozilla.com>
2008-02-08 00:35:56 +00:00
MALLOC_OUT:
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
if (nseen == false)
nreps = 1;
for (k = 0; k < nreps; k++) {
switch (opts[j]) {
case 'a':
opt_abort = false;
break;
case 'A':
opt_abort = true;
break;
case 'b':
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
opt_balance_threshold >>= 1;
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
break;
case 'B':
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
if (opt_balance_threshold == 0)
opt_balance_threshold = 1;
else if ((opt_balance_threshold << 1)
> opt_balance_threshold)
opt_balance_threshold <<= 1;
#endif
break;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
case 'c':
if (opt_cspace_max_2pow - 1 >
opt_qspace_max_2pow &&
opt_cspace_max_2pow >
CACHELINE_2POW)
opt_cspace_max_2pow--;
break;
case 'C':
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
if (opt_cspace_max_2pow < PAGE_SHIFT
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
- 1)
opt_cspace_max_2pow++;
break;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
case 'd':
#ifdef MALLOC_DSS
opt_dss = false;
#endif
break;
case 'D':
#ifdef MALLOC_DSS
opt_dss = true;
#endif
break;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
case 'f':
opt_dirty_max >>= 1;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
break;
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
case 'F':
if (opt_dirty_max == 0)
opt_dirty_max = 1;
else if ((opt_dirty_max << 1) != 0)
opt_dirty_max <<= 1;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
break;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
case 'g':
opt_mag = false;
break;
case 'G':
opt_mag = true;
break;
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
case 'j':
opt_junk = false;
break;
case 'J':
opt_junk = true;
break;
case 'k':
/*
* Chunks always require at least one
* header page, so chunks can never be
* smaller than two pages.
*/
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
if (opt_chunk_2pow > PAGE_SHIFT + 1)
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
opt_chunk_2pow--;
break;
case 'K':
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
if (opt_chunk_2pow + 1 <
(sizeof(size_t) << 3))
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
opt_chunk_2pow++;
break;
case 'm':
#ifdef MALLOC_DSS
opt_mmap = false;
#endif
break;
case 'M':
#ifdef MALLOC_DSS
opt_mmap = true;
#endif
break;
case 'n':
opt_narenas_lshift--;
break;
case 'N':
opt_narenas_lshift++;
break;
case 'p':
opt_print_stats = false;
break;
case 'P':
opt_print_stats = true;
break;
case 'q':
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
if (opt_qspace_max_2pow > QUANTUM_2POW)
opt_qspace_max_2pow--;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
break;
case 'Q':
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
if (opt_qspace_max_2pow + 1 <
opt_cspace_max_2pow)
opt_qspace_max_2pow++;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
break;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
case 'R':
if (opt_mag_size_2pow + 1 < (8U <<
SIZEOF_PTR_2POW))
opt_mag_size_2pow++;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
break;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
case 'r':
/*
* Make sure there's always at least
* one round per magazine.
*/
if ((1U << (opt_mag_size_2pow-1)) >=
sizeof(mag_t))
opt_mag_size_2pow--;
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
break;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
case 'u':
opt_utrace = false;
break;
case 'U':
opt_utrace = true;
break;
case 'v':
opt_sysv = false;
break;
case 'V':
opt_sysv = true;
break;
case 'x':
opt_xmalloc = false;
break;
case 'X':
opt_xmalloc = true;
break;
case 'z':
opt_zero = false;
break;
case 'Z':
opt_zero = true;
break;
default: {
char cbuf[2];
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
cbuf[0] = opts[j];
cbuf[1] = '\0';
_malloc_message(_getprogname(),
": (malloc) Unsupported character "
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
"in malloc options: '", cbuf,
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
"'\n");
}
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
}
}
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
/* Make sure that there is some method for acquiring memory. */
if (opt_dss == false && opt_mmap == false)
opt_mmap = true;
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Take care to call atexit() only once. */
if (opt_print_stats) {
/* Print statistics at exit. */
atexit(malloc_print_stats);
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_MAG
/*
* Calculate the actual number of rounds per magazine, taking into
* account header overhead.
*/
max_rounds = (1LLU << (opt_mag_size_2pow - SIZEOF_PTR_2POW)) -
(sizeof(mag_t) >> SIZEOF_PTR_2POW) + 1;
#endif
/* Set variables according to the value of opt_[qc]space_max_2pow. */
qspace_max = (1U << opt_qspace_max_2pow);
cspace_min = CACHELINE_CEILING(qspace_max);
if (cspace_min == qspace_max)
cspace_min += CACHELINE;
cspace_max = (1U << opt_cspace_max_2pow);
sspace_min = SUBPAGE_CEILING(cspace_max);
if (sspace_min == cspace_max)
sspace_min += SUBPAGE;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert(sspace_min < PAGE_SIZE);
sspace_max = PAGE_SIZE - SUBPAGE;
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
#ifdef MALLOC_TINY
assert(QUANTUM_2POW >= TINY_MIN_2POW);
#endif
assert(ntbins <= QUANTUM_2POW);
nqbins = qspace_max >> QUANTUM_2POW;
ncbins = ((cspace_max - cspace_min) >> CACHELINE_2POW) + 1;
nsbins = ((sspace_max - sspace_min) >> SUBPAGE_2POW) + 1;
nbins = ntbins + nqbins + ncbins + nsbins;
if (size2bin_init()) {
malloc_mutex_unlock(&init_lock);
return (true);
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Set variables according to the value of opt_chunk_2pow. */
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
chunksize = (1LU << opt_chunk_2pow);
chunksize_mask = chunksize - 1;
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
chunk_npages = (chunksize >> PAGE_SHIFT);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
{
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
size_t header_size;
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
/*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* Compute the header size such that it is large enough to
* contain the page map.
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
*/
header_size = sizeof(arena_chunk_t) +
Enhance arena_chunk_map_t to directly support run coalescing, and use the chunk map instead of red-black trees where possible. Remove the red-black trees and node objects that are obsoleted by this change. The net result is a ~1-2% memory savings, and a substantial allocation speed improvement.
2008-07-18 19:35:44 +00:00
(sizeof(arena_chunk_map_t) * (chunk_npages - 1));
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
arena_chunk_header_npages = (header_size >> PAGE_SHIFT) +
((header_size & PAGE_MASK) != 0);
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
}
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
arena_maxclass = chunksize - (arena_chunk_header_npages <<
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
PAGE_SHIFT);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(0, 0, 0);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef MALLOC_STATS
memset(&stats_chunks, 0, sizeof(chunk_stats_t));
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Various sanity checks that regard configuration. */
Use PAGE_{SIZE,MASK,SHIFT} from machine/param.h rather than hard-coding page size and using sysconf(3). Suggested by: marcel
2008-09-10 14:27:34 +00:00
assert(chunksize >= PAGE_SIZE);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Initialize chunks data. */
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_init(&huge_mtx);
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
extent_tree_ad_new(&huge);
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
malloc_mutex_init(&dss_mtx);
dss_base = sbrk(0);
dss_prev = dss_base;
dss_max = dss_base;
Add a separate tree to track arena chunks that contain dirty pages. This substantially improves worst case allocation performance, since O(lg n) tree search can be used instead of O(n) tree iteration. Use rb_wrap() instead of directly calling rb_*() macros.
2008-05-01 17:25:55 +00:00
extent_tree_szad_new(&dss_chunks_szad);
extent_tree_ad_new(&dss_chunks_ad);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#endif
#ifdef MALLOC_STATS
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
huge_nmalloc = 0;
huge_ndalloc = 0;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
huge_allocated = 0;
#endif
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
/* Initialize base allocation data structures. */
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
#ifdef MALLOC_STATS
Use extents rather than binary buddies to track free pages within chunks. This allows runs to be any multiple of the page size. The primary advantage is that large objects are no longer constrained to be 2^n pages, which can dramatically decrease internal fragmentation for large objects. This also allows the sizes for runs that back small objects to be more finely tuned. Free runs are searched for linearly using the chunk page map (with the help of some heuristic optimizations). This changes the allocation policy from "first best fit" to "first fit". A prototype red-black tree implementation for tracking free runs that implemented "first best fit" did not cause a measurable speed or memory usage difference for realistic chunk sizes (though of course it is possible to construct benchmarks that favor one allocation policy over another). Refine the handling of fullness constraints for small runs to be more tunable. Restructure the per chunk page map to contain only two fields per entry, rather than four. Also, increase each entry from 4 to 8 bytes, since it allows for 32-bit integers, without increasing the number of chunk header pages. Relax the maximum chunk size constraint. This is of no practical interest; it is merely fallout from the chunk page map restructuring. Revamp statistics gathering and reporting to be faster, clearer and more informative. Statistics gathering is fast enough now to have little to no impact on application speed, but it still requires approximately two extra pages of memory per arena (per process). This memory overhead may be acceptable for most systems, but we still need to leave statistics gathering disabled by default in RELENG branches. Rename NO_MALLOC_EXTRAS to MALLOC_PRODUCTION in order to make its intent clearer (i.e. it should be defined in RELENG branches).
2007-03-23 05:05:48 +00:00
base_mapped = 0;
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
#endif
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
#ifdef MALLOC_DSS
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
/*
Change the way base allocation is done for internal malloc data structures, in order to avoid the possibility of attempted recursive lock acquisition for chunks_mtx. Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>
2006-09-08 17:52:15 +00:00
* Allocate a base chunk here, since it doesn't actually have to be
* chunk-aligned. Doing this before allocating any other chunks allows
* the use of space that would otherwise be wasted.
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
*/
Add the 'D' and 'M' run time options, and use them to control whether memory is acquired from the system via sbrk(2) and/or mmap(2). By default, use sbrk(2) only, in order to support traditional use of resource limits. Additionally, when both options are enabled, prefer the data segment to anonymous mappings, in order to coexist better with large file mappings in applications on 32-bit platforms. This change has the potential to increase memory fragmentation due to the linear nature of the data segment, but from a performance perspective this is mitigated by the use of madvise(2). [1] Add the ability to interpret integer prefixes in MALLOC_OPTIONS processing. For example, MALLOC_OPTIONS=lllllllll can now be specified as MALLOC_OPTIONS=9l. Reported by: [1] rwatson Design review: [1] alc, peter, rwatson
2007-12-27 23:29:44 +00:00
if (opt_dss)
base_pages_alloc(0);
Add USE_BRK-specific code in malloc_init_hard() to allow the first internally used chunk to start at the beginning of the heap, rather than at a chunk-aligned address. This reduces mapped memory somewhat for 32-bit architectures. Add the arena_run_link_t type and use it wherever a run object is only used as a ring 'header'. This saves approximately 40 kB of memory per arena. Remove an obsolete (no longer used) code path from base_alloc(), which supported the internal allocation of objects larger than the chunk size. Enhance chunk_dealloc() to cache chunk addresses for all deallocated chunks. This has no impact for most programs, but has the potential to reduce VM map fragmentation for programs that use huge allocations.
2006-03-24 00:28:08 +00:00
#endif
Track dirty unused pages so that they can be purged if they exceed a threshold, according to the 'F' MALLOC_OPTIONS flag. This obsoletes the 'H' flag. Try to realloc() large objects in place. This substantially speeds up incremental large reallocations in the common case. Fix a bug in arena_ralloc() that caused relocation of sub-page objects even if the old and new sizes were in the same size class. Maintain trees of runs and simplify the per-chunk page map. This allows logarithmic-time searching for sufficiently large runs in arena_run_alloc(), whereas the previous algorithm required linear time in the worst case. Break various large functions into smaller sub-functions, and inline only the functions that are in the fast path for small object allocation/deallocation. Remove an unnecessary check in base_pages_alloc_mmap(). Avoid integer division in choose_arena() for the NO_TLS case on single-CPU systems.
2008-02-06 02:59:54 +00:00
base_nodes = NULL;
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_init(&base_mtx);
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (ncpus > 1) {
Add comments and reformat/rearrange code. There are no significant functional changes in this commit.
2006-03-26 23:37:25 +00:00
/*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* For SMP systems, create twice as many arenas as there are
* CPUs by default.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
opt_narenas_lshift++;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
``phkmalloc'' Performance is comparable to gnumalloc if you have sufficient RAM, and it screams around it if you don't. Compiled with "EXTRA_SANITY" until further notice. see malloc.3 for more details.
1995-09-16 09:28:13 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Determine how many arenas to use. */
Fix calculation of the number of arenas to use on multi-processor systems.
2006-02-04 01:11:30 +00:00
narenas = ncpus;
Allow the 'n' option to decrease the number of arenas below the default, to as little as one arena. Also, limit the number of arenas to avoid a potential invariant violation in base_alloc().
2006-03-26 23:41:35 +00:00
if (opt_narenas_lshift > 0) {
if ((narenas << opt_narenas_lshift) > narenas)
narenas <<= opt_narenas_lshift;
/*
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
* Make sure not to exceed the limits of what base_alloc() can
* handle.
Allow the 'n' option to decrease the number of arenas below the default, to as little as one arena. Also, limit the number of arenas to avoid a potential invariant violation in base_alloc().
2006-03-26 23:41:35 +00:00
*/
Fix posix_memalign() for large objects. Now that runs are extents rather than binary buddies, the alignment guarantees are weaker, which requires a more complex aligned allocation algorithm, similar to that used for alignment greater than the chunk size. Reported by: matteo
2007-03-23 22:58:15 +00:00
if (narenas * sizeof(arena_t *) > chunksize)
narenas = chunksize / sizeof(arena_t *);
Allow the 'n' option to decrease the number of arenas below the default, to as little as one arena. Also, limit the number of arenas to avoid a potential invariant violation in base_alloc().
2006-03-26 23:41:35 +00:00
} else if (opt_narenas_lshift < 0) {
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
if ((narenas >> -opt_narenas_lshift) < narenas)
narenas >>= -opt_narenas_lshift;
Allow the 'n' option to decrease the number of arenas below the default, to as little as one arena. Also, limit the number of arenas to avoid a potential invariant violation in base_alloc().
2006-03-26 23:41:35 +00:00
/* Make sure there is at least one arena. */
if (narenas == 0)
narenas = 1;
}
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
assert(narenas != 0);
for (narenas_2pow = 0;
(narenas >> (narenas_2pow + 1)) != 0;
narenas_2pow++);
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
#ifdef NO_TLS
if (narenas > 1) {
static const unsigned primes[] = {1, 3, 5, 7, 11, 13, 17, 19,
23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83,
89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149,
151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211,
223, 227, 229, 233, 239, 241, 251, 257, 263};
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
unsigned nprimes, parenas;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Pick a prime number of hash arenas that is more than narenas
* so that direct hashing of pthread_self() pointers tends to
* spread allocations evenly among the arenas.
*/
assert((narenas & 1) == 0); /* narenas must be even. */
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
nprimes = (sizeof(primes) >> SIZEOF_INT_2POW);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
parenas = primes[nprimes - 1]; /* In case not enough primes. */
for (i = 1; i < nprimes; i++) {
if (primes[i] > narenas) {
parenas = primes[i];
break;
}
}
narenas = parenas;
}
#endif
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifndef NO_TLS
# ifndef MALLOC_BALANCE
next_arena = 0;
# endif
#endif
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Allocate and initialize arenas. */
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
arenas = (arena_t **)base_alloc(sizeof(arena_t *) * narenas);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
if (arenas == NULL) {
malloc_mutex_unlock(&init_lock);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (true);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Zero the array. In practice, this should always be pre-zeroed,
* since it was just mmap()ed, but let's be sure.
*/
memset(arenas, 0, sizeof(arena_t *) * narenas);
phkmalloc/2 "zero' and 'junk' options to help find and diagnose malloc abuse. EXTRA_SANITY defaults "junk" to on. Don't dump the internal state, unless EXTRA_SANITY. General code cleanup. Error messages cleaned up a bit, more checking earlier. EXTRA_SANITY is default at this time (just in case). Performance (without EXTRA_SANITY) is better, beats gnumalloc in both time & space most of the time: # In-memory test. ./malloc 50000000 2000 8192 159.2u 1.5s 2:41.85 99.3% 5+7742k 0+0io 0pf+0w ./gnumalloc 50000000 2000 8192 272.6u 0.4s 4:35.01 99.3% 5+8533k 0+0io 0pf+0w # Swap-space test. ./malloc 500000 14000 8192 6.5u 4.1s 4:08.87 4.3% 5+49209k 0+0io 9772pf+0w ./gnumalloc 500000 14000 8192 16.2u 14.5s 15:36.14 3.2% 5+54100k 0+0io 47651pf+0w # Small items test. ./malloc 20000000 20000 2048 67.0u 0.3s 1:07.83 99.2% 5+18199k 0+0io 4pf+0w ./gnumalloc 20000000 20000 2048 66.2u 0.3s 1:07.03 99.3% 5+18107k 0+0io 0pf+0w SANITY is not an option anymore. (!!)
1995-10-08 18:44:20 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* Initialize one arena here. The rest are lazily created in
Fix stats printing of the amount of memory currently consumed by huge allocations. [1] Fix calculation of the number of arenas when 'n' is specified via MALLOC_OPTIONS. Clean up various style inconsistencies. Obtained from: [1] NetBSD
2007-11-27 03:09:23 +00:00
* choose_arena_hard().
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
arenas_extend(0);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
if (arenas[0] == NULL) {
malloc_mutex_unlock(&init_lock);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (true);
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
}
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
#ifndef NO_TLS
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
/*
* Assign the initial arena to the initial thread, in order to avoid
* spurious creation of an extra arena if the application switches to
* threaded mode.
*/
arenas_map = arenas[0];
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#endif
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
/*
* Seed here for the initial thread, since choose_arena_hard() is only
Remove support for lazy deallocation. Benchmarks across a wide range of allocation patterns, number of CPUs, and MALLOC_OPTIONS settings indicate that lazy deallocation has the potential to worsen throughput dramatically. Performance degradation occurs when multiple threads try to clear the lazy free cache simultaneously. Various experiments to avoid this bottleneck failed to completely solve this problem, while adding yet more complexity.
2008-02-17 17:09:24 +00:00
* called for other threads. The seed value doesn't really matter.
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
*/
Refactor features a bit in order to make it possible to disable lazy deallocation and dynamic load balancing via the MALLOC_LAZY_FREE and MALLOC_BALANCE knobs. This is a non-functional change, since these features are still enabled when possible. Clean up a few things that more pedantic compiler settings would cause complaints over.
2007-12-17 01:20:04 +00:00
#ifdef MALLOC_BALANCE
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
SPRN(balance, 42);
Implement lazy deallocation of small objects. For each arena, maintain a vector of slots for lazily freed objects. For each deallocation, before doing the hard work of locking the arena and deallocating, try several times to randomly insert the object into the vector using atomic operations. This approach is particularly effective at reducing contention for multi-threaded applications that use the producer-consumer model, wherein one producer thread allocates objects, then multiple consumer threads deallocate those objects.
2007-11-27 03:13:15 +00:00
#endif
Isolated all the crap for thread-safe so I can see what goes on again...
1996-07-03 05:03:07 +00:00
Implement dynamic load balancing of thread-->arena mapping, based on lock contention. The intent is to dynamically adjust to load imbalances, which can cause severe contention. Use pthread mutexes where possible instead of libc "spinlocks" (they aren't actually spin locks). Conceptually, this change is meant only to support the dynamic load balancing code by enabling the use of spin locks, but it has the added apparent benefit of substantially improving performance due to reduced context switches when there is moderate arena lock contention. Proper tuning parameter configuration for this change is a finicky business, and it is very much machine-dependent. One seemingly promising solution would be to run a tuning program during operating system installation that computes appropriate settings for load balancing. (The pthreads adaptive spin locks should probably be similarly tuned.)
2007-11-27 03:17:30 +00:00
malloc_spin_init(&arenas_lock);
Rearrange (centralize) initialization of mallocs internals to always be done before the first call, even if this is a malloc(0) call. PR: 62859
2004-03-07 20:41:27 +00:00
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
malloc_initialized = true;
Add init_lock, and use it to protect against allocator initialization races. This isn't currently necessary for libpthread or libthr, but without it external threads libraries like the linuxthreads port are not safe to use. Reported by: ganbold@micom.mng.net
2006-04-04 19:46:28 +00:00
malloc_mutex_unlock(&init_lock);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
return (false);
Isolated all the crap for thread-safe so I can see what goes on again...
1996-07-03 05:03:07 +00:00
}
Integrate calloc with the rest of the gang. Various portability and stylistic cleanups. Kill MALLOC_STATS & the 'D' option. Fix the 'V' option. Major overhaul of the man-page. You milage should not vary. Reviewed by: Keith Bostic <bostic@bostic.com> Submitted by: Keith Bostic <bostic@bostic.com>
1997-06-22 17:54:27 +00:00
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
/*
Modify allocation policy, in order to avoid excessive fragmentation for allocation patterns that involve a relatively even mixture of many different size classes. Reduce the chunk size from 16 MB to 2 MB. Since chunks are now carved up using an address-ordered first best fit policy, VM map fragmentation is much less likely, which makes smaller chunks not as much of a risk. This reduces the virtual memory size of most applications. Remove redzones, since program buffer overruns are no longer as likely to corrupt malloc data structures. Remove the C MALLOC_OPTIONS flag, and add H and S.
2006-03-17 09:00:27 +00:00
* End general internal functions.
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*/
/******************************************************************************/
/*
* Begin malloc(3)-compatible functions.
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
*/
void *
malloc(size_t size)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
if (malloc_init()) {
ret = NULL;
goto RETURN;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (size == 0) {
if (opt_sysv == false)
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
size = 1;
else {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
ret = NULL;
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
goto RETURN;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = imalloc(size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
RETURN:
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in malloc(): out of memory\n", "",
"");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
abort();
}
errno = ENOMEM;
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(0, size, ret);
return (ret);
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
int
posix_memalign(void **memptr, size_t alignment, size_t size)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
int ret;
void *result;
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
if (malloc_init())
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
result = NULL;
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
else {
/* Make sure that alignment is a large enough power of 2. */
if (((alignment - 1) & alignment) != 0
|| alignment < sizeof(void *)) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in posix_memalign(): "
"invalid alignment\n", "", "");
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
abort();
}
result = NULL;
ret = EINVAL;
goto RETURN;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
result = ipalloc(alignment, size);
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
if (result == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in posix_memalign(): out of memory\n",
"", "");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
abort();
}
ret = ENOMEM;
goto RETURN;
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
*memptr = result;
ret = 0;
RETURN:
UTRACE(0, size, result);
return (ret);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
void *
calloc(size_t num, size_t size)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
size_t num_size;
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
if (malloc_init()) {
Fix a utrace(2)-related bug in calloc(3). Integrate various pedantic cleanups. Submitted by: Andrew Doran <ad@netbsd.org>
2007-01-31 22:54:19 +00:00
num_size = 0;
Return NULL if there is an OOM error during initialization, rather than allowing the error to be fatal. Move a label in order to make sure to properly handle errors in malloc(0). Reported by: Alastair D'Silva, Saneto Takanori
2006-01-19 02:11:05 +00:00
ret = NULL;
goto RETURN;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
num_size = num * size;
if (num_size == 0) {
Correctly handle the case in calloc(num, size) where (size_t)(num * size) == 0 but both num and size are nonzero. Reported by: Ilja van Sprundel Approved by: jasone Security: Integer overflow; calloc was allocating 1 byte in response to a request for a multiple of 2^32 (or 2^64) bytes instead of returning NULL.
2006-08-13 21:54:47 +00:00
if ((opt_sysv == false) && ((num == 0) || (size == 0)))
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
num_size = 1;
else {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
ret = NULL;
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
goto RETURN;
}
Refactor per-run bitmap manipulation functions so that bitmap offsets only have to be calculated once per allocator operation. Make nil const. Update various comments. Remove/avoid division where possible. For the one division operation that remains in the critical path, add a switch statement that has a case for each small size class, and do division with a constant divisor in each case. This allows the compiler to generate optimized code that does not use hardware division [1]. Obtained from: peter [1]
2006-04-04 03:51:47 +00:00
/*
* Try to avoid division here. We know that it isn't possible to
* overflow during multiplication if neither operand uses any of the
* most significant half of the bits in a size_t.
*/
} else if (((num | size) & (SIZE_T_MAX << (sizeof(size_t) << 2)))
&& (num_size / size != num)) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* size_t overflow. */
ret = NULL;
goto RETURN;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
ret = icalloc(num_size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
RETURN:
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in calloc(): out of memory\n", "",
"");
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
abort();
}
errno = ENOMEM;
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Optimize runtime performance, primary using the following techniques: * Avoid choosing an arena until it's certain that an arena is needed for allocation. * Convert division/multiplication to bitshifting where possible. * Avoid accessing TLS variables in single-threaded code. * Reduce the amount of pointer dereferencing. * Move lock acquisition in critical paths to only protect the the code that requires synchronization, and completely remove locking where possible.
2006-03-30 20:25:52 +00:00
UTRACE(0, num_size, ret);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
return (ret);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *
realloc(void *ptr, size_t size)
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void *ret;
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (size == 0) {
if (opt_sysv == false)
size = 1;
else {
if (ptr != NULL)
idalloc(ptr);
ret = NULL;
goto RETURN;
}
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ptr != NULL) {
assert(malloc_initialized);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
ret = iralloc(ptr, size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in realloc(): out of "
"memory\n", "", "");
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
abort();
Optimize realloc() to reallocate in place if the old and new sizes are close enough to each other that reallocation would allocate a new region of the same size. This improves the performance of repeated incremental reallocations by up to three orders of magnitude. [1] Fix arena_new() to properly constrain run size if a small chunk size was specified during runtime configuration. Suggested by: se [1]
2006-03-19 18:28:06 +00:00
}
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
errno = ENOMEM;
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
} else {
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (malloc_init())
ret = NULL;
else
ret = imalloc(size);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ret == NULL) {
if (opt_xmalloc) {
Avoid using vsnprintf(3) unless MALLOC_STATS is defined, in order to avoid substantial potential bloat for static binaries that do not otherwise use any printf(3)-family functions. [1] Rearrange arena_run_t so that the region bitmask can be minimally sized according to constraints related to each bin's size class. Previously, the region bitmask was the same size for all run headers, which wasted a measurable amount of memory. Rather than making runs for small objects as large as possible, make runs as small as possible such that header overhead stays below a certain bound. There are two exceptions that override the header overhead bound: 1) If the bound is impossible to honor, it is relaxed on a per-size-class basis. Since there is one bit of header overhead per object (plus a constant), it is impossible to achieve a header overhead less than or equal to 1/(# of bits per object). For the current setting of maximum 0.5% header overhead, this relaxation comes into play for {2, 4, 8, 16}-byte objects, for which header overhead is (on 64-bit systems) {7.1, 4.3, 2.2, 1.2}%, respectively. 2) There is still a cap on small run size, still set to 64kB. This comes into play for {1024, 2048}-byte objects, for which header overhead is {1.6, 3.1}%, respectively. In practice, this reduces the run sizes, which makes worst case low-water memory usage due to fragmentation less bad. It also reduces worst case high-water run fragmentation due to non-full runs, but this is only a constant improvement (most important to small short-lived processes). Reduce the default chunk size from 2MB to 1MB. Benchmarks indicate that the external fragmentation reduction makes 1MB the new sweet spot (as small as possible without adversely affecting performance). Reported by: [1] kientzle
2007-03-20 03:44:10 +00:00
_malloc_message(_getprogname(),
": (malloc) Error in realloc(): out of "
"memory\n", "", "");
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
abort();
}
errno = ENOMEM;
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
RETURN:
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(ptr, size, ret);
return (ret);
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
void
free(void *ptr)
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
UTRACE(ptr, 0, 0);
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
if (ptr != NULL) {
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
assert(malloc_initialized);
idalloc(ptr);
}
Remove the triplicity in the public functions by vectoring them all through a realloc like function. Make the malloc_active variable a local static to this new function. Don't warn about recursion more than once per base call. constify malloc_func.
2004-02-21 09:14:38 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* End malloc(3)-compatible functions.
*/
/******************************************************************************/
Add malloc_usable_size(3). Discussed with: arch@
2006-03-28 22:16:04 +00:00
/*
* Begin non-standard functions.
*/
size_t
malloc_usable_size(const void *ptr)
{
assert(ptr != NULL);
Make the behavior of malloc(0) standards-compliant by getting rid of nil, and instead creating a small allocation for each malloc(0) call. The optional SysV compatibility behavior remains unchanged. Add a couple of assertions. Fix a couple of typos in error message strings.
2006-06-30 20:54:15 +00:00
return (isalloc(ptr));
Add malloc_usable_size(3). Discussed with: arch@
2006-03-28 22:16:04 +00:00
}
/*
* End non-standard functions.
*/
/******************************************************************************/
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
/*
Add thread-specific caching for small size classes, based on magazines. This caching allows for completely lock-free allocation/deallocation in the steady state, at the expense of likely increased memory use and fragmentation. Reduce the default number of arenas to 2*ncpus, since thread-specific caching typically reduces arena contention. Modify size class spacing to include ranges of 2^n-spaced, quantum-spaced, cacheline-spaced, and subpage-spaced size classes. The advantages are: fewer size classes, reduced false cacheline sharing, and reduced internal fragmentation for allocations that are slightly over 512, 1024, etc. Increase RUN_MAX_SMALL, in order to limit fragmentation for the subpage-spaced size classes. Add a size-->bin lookup table for small sizes to simplify translating sizes to size classes. Include a hard-coded constant table that is used unless custom size class spacing is specified at run time. Add the ability to disable tiny size classes at compile time via MALLOC_TINY.
2008-08-27 02:00:53 +00:00
* Begin library-private functions.
*/
/******************************************************************************/
/*
* Begin thread cache.
*/
/*
* We provide an unpublished interface in order to receive notifications from
* the pthreads library whenever a thread exits. This allows us to clean up
* thread caches.
*/
void
_malloc_thread_cleanup(void)
{
#ifdef MALLOC_MAG
if (mag_rack != NULL) {
assert(mag_rack != (void *)-1);
mag_rack_destroy(mag_rack);
#ifdef MALLOC_DEBUG
mag_rack = (void *)-1;
#endif
}
#endif
}
/*
* The following functions are used by threading libraries for protection of
* malloc during fork(). These functions are only called if the program is
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
* running in threaded mode, so there is no need to check whether the program
* is threaded here.
*/
void
_malloc_prefork(void)
{
Fix a lock order reversal bug that could cause deadlock during fork(2). Reported by: kib
2008-12-01 10:20:59 +00:00
bool again;
unsigned i, j;
arena_t *larenas[narenas], *tarenas[narenas];
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Acquire all mutexes in a safe order. */
Fix a lock order reversal bug that could cause deadlock during fork(2). Reported by: kib
2008-12-01 10:20:59 +00:00
/*
* arenas_lock must be acquired after all of the arena mutexes, in
* order to avoid potential deadlock with arena_lock_balance[_hard]().
* Since arenas_lock protects the arenas array, the following code has
* to race with arenas_extend() callers until it succeeds in locking
* all arenas before locking arenas_lock.
*/
memset(larenas, 0, sizeof(arena_t *) * narenas);
do {
again = false;
malloc_spin_lock(&arenas_lock);
for (i = 0; i < narenas; i++) {
if (arenas[i] != larenas[i]) {
memcpy(tarenas, arenas, sizeof(arena_t *) *
narenas);
malloc_spin_unlock(&arenas_lock);
for (j = 0; j < narenas; j++) {
if (larenas[j] != tarenas[j]) {
larenas[j] = tarenas[j];
malloc_spin_lock(
&larenas[j]->lock);
}
}
again = true;
break;
}
}
} while (again);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_lock(&base_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
malloc_mutex_lock(&huge_mtx);
#ifdef MALLOC_DSS
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_lock(&dss_mtx);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
#endif
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
void
_malloc_postfork(void)
{
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
unsigned i;
Fix a lock order reversal bug that could cause deadlock during fork(2). Reported by: kib
2008-12-01 10:20:59 +00:00
arena_t *larenas[narenas];
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/* Release all mutexes, now that fork() has completed. */
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
#ifdef MALLOC_DSS
Fix a major chunk-related memory leak in chunk_dealloc_dss_record(). [1] Clean up DSS-related locking and protect all pertinent variables with dss_mtx (remove dss_chunks_mtx). This fixes race conditions that could cause chunk leaks. Reported by: [1] kris
2007-12-31 06:19:48 +00:00
malloc_mutex_unlock(&dss_mtx);
Fix a bug related to sbrk() calls that could cause address space leaks. This is a long-standing bug, but until recent changes it was difficult to trigger, and even then its impact was non-catastrophic, with the exception of revision 1.157. Optimize chunk_alloc_mmap() to avoid the need for unmapping pages in the common case. Thanks go to Kris Kennaway for a patch that inspired this change. Do not maintain a record of previously mmap'ed chunk address ranges. The original intent was to avoid the extra system call overhead in chunk_alloc_mmap(), which is no longer a concern. This also allows some simplifications for the tree of unused DSS chunks. Introduce huge_mtx and dss_chunks_mtx to replace chunks_mtx. There was no compelling reason to use the same mutex for these disjoint purposes. Avoid memset() for huge allocations when possible. Maintain two trees instead of one for tracking unused DSS address ranges. This allows scalable allocation of multi-chunk huge objects in the DSS. Previously, multi-chunk huge allocation requests failed if the DSS could not be extended.
2007-12-31 00:59:16 +00:00
#endif
malloc_mutex_unlock(&huge_mtx);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
Add a separate simple internal base allocator and remove base_arena, so that there is never any need to recursively call the main allocation functions. Remove recursive spinlock support, since it is no longer needed. Allow chunks to be as small as the page size. Correctly propagate OOM errors from arena_new().
2006-01-16 05:13:49 +00:00
malloc_mutex_unlock(&base_mtx);
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
Fix a lock order reversal bug that could cause deadlock during fork(2). Reported by: kib
2008-12-01 10:20:59 +00:00
memcpy(larenas, arenas, sizeof(arena_t *) * narenas);
malloc_spin_unlock(&arenas_lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
for (i = 0; i < narenas; i++) {
Fix a lock order reversal bug that could cause deadlock during fork(2). Reported by: kib
2008-12-01 10:20:59 +00:00
if (larenas[i] != NULL)
malloc_spin_unlock(&larenas[i]->lock);
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
}
In preparation for a new malloc implementation: * Add posix_memalign(). * Move calloc() from calloc.c to malloc.c. Add a calloc() implementation in rtld-elf in order to make the loader happy (even though calloc() isn't used in rtld-elf). * Add _malloc_prefork() and _malloc_postfork(), and use them instead of directly manipulating __malloc_lock. Approved by: phk, markm (mentor)
2006-01-12 07:28:21 +00:00
}
Replace malloc(), calloc(), posix_memalign(), realloc(), and free() with a scalable concurrent allocator implementation. Reviewed by: current@ Approved by: phk, markm (mentor)
2006-01-13 18:38:56 +00:00
/*
* End library-private functions.
*/
/******************************************************************************/
Reference in New Issue Copy Permalink