2003-08-12 23:24:05 +00:00
|
|
|
/*-
|
2017-11-20 19:43:44 +00:00
|
|
|
* SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
|
*
|
2003-08-12 23:24:05 +00:00
|
|
|
* Copyright (c) 1998 Matthew Dillon. All Rights Reserved.
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2016-09-15 13:16:20 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
2003-08-12 23:24:05 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
|
* without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS
|
|
|
|
|
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
|
|
|
|
|
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
|
|
|
|
|
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
|
|
|
|
* GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
|
|
|
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
|
|
|
|
|
* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
|
|
|
|
|
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
|
|
|
|
|
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
|
*/
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* BLIST.C - Bitmap allocator/deallocator, using a radix tree with hinting
|
|
|
|
|
*
|
|
|
|
|
* This module implements a general bitmap allocator/deallocator. The
|
2017-07-17 23:16:33 +00:00
|
|
|
* allocator eats around 2 bits per 'block'. The module does not
|
|
|
|
|
* try to interpret the meaning of a 'block' other than to return
|
1999-01-21 08:11:06 +00:00
|
|
|
* SWAPBLK_NONE on an allocation failure.
|
|
|
|
|
*
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* A radix tree controls access to pieces of the bitmap, and includes
|
|
|
|
|
* auxiliary information at each interior node about the availabilty of
|
2020-07-25 18:29:10 +00:00
|
|
|
* contiguous free blocks in the subtree rooted at that node. A radix
|
|
|
|
|
* constant defines the size of the bitmaps contained in a leaf node
|
|
|
|
|
* and the number of descendents of each of the meta (interior) nodes.
|
|
|
|
|
* Each subtree is associated with a range of blocks. The root of any
|
|
|
|
|
* subtree stores a hint field that defines an upper bound on the size
|
|
|
|
|
* of the largest allocation that can begin in the associated block
|
|
|
|
|
* range. A hint is an upper bound on a potential allocation, but not
|
|
|
|
|
* necessarily a tight upper bound.
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* The bitmap field in each node directs the search for available blocks.
|
|
|
|
|
* For a leaf node, a bit is set if the corresponding block is free. For a
|
|
|
|
|
* meta node, a bit is set if the corresponding subtree contains a free
|
|
|
|
|
* block somewhere within it. The search at a meta node considers only
|
|
|
|
|
* children of that node that represent a range that includes a free block.
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
|
|
|
|
* The hinting greatly increases code efficiency for allocations while
|
|
|
|
|
* the general radix structure optimizes both allocations and frees. The
|
2017-07-17 23:16:33 +00:00
|
|
|
* radix tree should be able to operate well no matter how much
|
1999-01-21 08:11:06 +00:00
|
|
|
* fragmentation there is and no matter how large a bitmap is used.
|
|
|
|
|
*
|
2013-02-05 20:08:33 +00:00
|
|
|
* The blist code wires all necessary memory at creation time. Neither
|
|
|
|
|
* allocations nor frees require interaction with the memory subsystem.
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* The non-blocking nature of allocations and frees is required by swap
|
|
|
|
|
* code (vm/swap_pager.c).
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* LAYOUT: The radix tree is laid out recursively using a linear array.
|
|
|
|
|
* Each meta node is immediately followed (laid out sequentially in
|
2020-07-25 18:29:10 +00:00
|
|
|
* memory) by BLIST_RADIX lower-level nodes. This is a recursive
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* structure but one that can be easily scanned through a very simple
|
|
|
|
|
* 'skip' calculation. The memory allocation is only large enough to
|
|
|
|
|
* cover the number of blocks requested at creation time. Nodes that
|
|
|
|
|
* represent blocks beyond that limit, nodes that would never be read
|
|
|
|
|
* or written, are not allocated, so that the last of the
|
2020-07-25 18:29:10 +00:00
|
|
|
* BLIST_RADIX lower-level nodes of a some nodes may not be allocated.
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
2017-07-17 23:16:33 +00:00
|
|
|
* NOTE: the allocator cannot currently allocate more than
|
2020-07-25 18:29:10 +00:00
|
|
|
* BLIST_RADIX blocks per call. It will panic with 'allocation too
|
2017-07-17 23:16:33 +00:00
|
|
|
* large' if you try. This is an area that could use improvement. The
|
|
|
|
|
* radix is large enough that this restriction does not effect the swap
|
|
|
|
|
* system, though. Currently only the allocation code is affected by
|
1999-01-21 08:11:06 +00:00
|
|
|
* this algorithmic unfeature. The freeing code can handle arbitrary
|
|
|
|
|
* ranges.
|
|
|
|
|
*
|
|
|
|
|
* This code can be compiled stand-alone for debugging.
|
|
|
|
|
*/
|
|
|
|
|
|
2003-06-11 00:56:59 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifdef _KERNEL
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
#include <sys/param.h>
|
|
|
|
|
#include <sys/systm.h>
|
|
|
|
|
#include <sys/lock.h>
|
|
|
|
|
#include <sys/kernel.h>
|
|
|
|
|
#include <sys/blist.h>
|
|
|
|
|
#include <sys/malloc.h>
|
2017-09-10 17:46:03 +00:00
|
|
|
#include <sys/sbuf.h>
|
2001-07-04 16:20:28 +00:00
|
|
|
#include <sys/proc.h>
|
2017-07-17 23:16:33 +00:00
|
|
|
#include <sys/mutex.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
#else
|
|
|
|
|
|
|
|
|
|
#ifndef BLIST_NO_DEBUG
|
|
|
|
|
#define BLIST_DEBUG
|
|
|
|
|
#endif
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
#include <sys/errno.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
#include <sys/types.h>
|
2017-06-07 16:04:34 +00:00
|
|
|
#include <sys/malloc.h>
|
2017-09-10 17:46:03 +00:00
|
|
|
#include <sys/sbuf.h>
|
2019-05-10 18:22:40 +00:00
|
|
|
#include <assert.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
#include <stdio.h>
|
|
|
|
|
#include <string.h>
|
The blst_radix_init function has two purposes - to compute the number of
nodes to allocate for the blist, and to initialize them. The computation
can be done much more quickly by identifying the terminating node, if any,
at every level of the tree and then summing the number of nodes at each
level that precedes the topmost terminator. The initialization can also be
done quickly, since settings at the root mark the tree as all-allocated, and
only a few terminator nodes need to be marked in the rest of the tree.
Eliminate blst_radix_init, and perform its two functions more simply in
blist_create.
The allocation of the blist takes places in two pieces, but there's no good
reason to do so, when a single allocation is sufficient, and simpler.
Allocate the blist struct, and the array of nodes associated with it, with a
single allocation.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11968
2017-10-08 22:17:39 +00:00
|
|
|
#include <stddef.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
#include <stdlib.h>
|
|
|
|
|
#include <stdarg.h>
|
2017-06-27 17:45:26 +00:00
|
|
|
#include <stdbool.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
#define bitcount64(x) __bitcount64((uint64_t)(x))
|
2002-12-15 19:17:57 +00:00
|
|
|
#define malloc(a,b,c) calloc(a, 1)
|
1999-01-21 08:11:06 +00:00
|
|
|
#define free(a,b) free(a)
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
#define ummin(a,b) ((a) < (b) ? (a) : (b))
|
2019-07-06 06:15:03 +00:00
|
|
|
#define imin(a,b) ((a) < (b) ? (a) : (b))
|
2019-05-10 18:22:40 +00:00
|
|
|
#define KASSERT(a,b) assert(a)
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
#include <sys/blist.h>
|
|
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* static support functions
|
|
|
|
|
*/
|
2019-05-11 16:15:13 +00:00
|
|
|
static daddr_t blst_leaf_alloc(blmeta_t *scan, daddr_t blk,
|
|
|
|
|
int *count, int maxcount);
|
|
|
|
|
static daddr_t blst_meta_alloc(blmeta_t *scan, daddr_t cursor, int *count,
|
|
|
|
|
int maxcount, u_daddr_t radix);
|
1999-01-21 08:11:06 +00:00
|
|
|
static void blst_leaf_free(blmeta_t *scan, daddr_t relblk, int count);
|
2017-07-17 23:16:33 +00:00
|
|
|
static void blst_meta_free(blmeta_t *scan, daddr_t freeBlk, daddr_t count,
|
2017-08-13 16:39:49 +00:00
|
|
|
u_daddr_t radix);
|
2017-07-17 23:16:33 +00:00
|
|
|
static void blst_copy(blmeta_t *scan, daddr_t blk, daddr_t radix,
|
2017-08-01 03:51:26 +00:00
|
|
|
blist_t dest, daddr_t count);
|
2017-06-09 16:19:24 +00:00
|
|
|
static daddr_t blst_leaf_fill(blmeta_t *scan, daddr_t blk, int count);
|
|
|
|
|
static daddr_t blst_meta_fill(blmeta_t *scan, daddr_t allocBlk, daddr_t count,
|
2017-08-13 16:39:49 +00:00
|
|
|
u_daddr_t radix);
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifndef _KERNEL
|
2017-06-27 17:45:26 +00:00
|
|
|
static void blst_radix_print(blmeta_t *scan, daddr_t blk, daddr_t radix,
|
2017-08-01 03:51:26 +00:00
|
|
|
int tab);
|
1999-01-21 08:11:06 +00:00
|
|
|
#endif
|
|
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifdef _KERNEL
|
1999-01-21 08:11:06 +00:00
|
|
|
static MALLOC_DEFINE(M_SWAP, "SWAP", "Swap space");
|
|
|
|
|
#endif
|
|
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
#define BLIST_MASK (BLIST_RADIX - 1)
|
2017-08-04 04:23:23 +00:00
|
|
|
|
2017-08-01 03:51:26 +00:00
|
|
|
/*
|
|
|
|
|
* For a subtree that can represent the state of up to 'radix' blocks, the
|
2020-07-25 18:29:10 +00:00
|
|
|
* number of leaf nodes of the subtree is L=radix/BLIST_RADIX. If 'm'
|
|
|
|
|
* is short for BLIST_RADIX, then for a tree of height h with L=m**h
|
2017-08-01 03:51:26 +00:00
|
|
|
* leaf nodes, the total number of tree nodes is 1 + m + m**2 + ... + m**h,
|
|
|
|
|
* or, equivalently, (m**(h+1)-1)/(m-1). This quantity is called 'skip'
|
|
|
|
|
* in the 'meta' functions that process subtrees. Since integer division
|
|
|
|
|
* discards remainders, we can express this computation as
|
|
|
|
|
* skip = (m * m**h) / (m - 1)
|
2020-07-25 18:29:10 +00:00
|
|
|
* skip = (m * (radix / m)) / (m - 1)
|
|
|
|
|
* skip = radix / (m - 1)
|
2017-08-04 04:23:23 +00:00
|
|
|
* so that simple integer division by a constant can safely be used for the
|
|
|
|
|
* calculation.
|
2017-08-01 03:51:26 +00:00
|
|
|
*/
|
|
|
|
|
static inline daddr_t
|
|
|
|
|
radix_to_skip(daddr_t radix)
|
|
|
|
|
{
|
|
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
return (radix / BLIST_MASK);
|
2017-09-10 17:46:03 +00:00
|
|
|
}
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
/*
|
|
|
|
|
* Provide a mask with count bits set, starting as position n.
|
|
|
|
|
*/
|
|
|
|
|
static inline u_daddr_t
|
|
|
|
|
bitrange(int n, int count)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (((u_daddr_t)-1 << n) &
|
2020-07-25 18:29:10 +00:00
|
|
|
((u_daddr_t)-1 >> (BLIST_RADIX - (n + count))));
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
}
|
|
|
|
|
|
2017-09-10 17:46:03 +00:00
|
|
|
/*
|
2019-05-11 09:09:10 +00:00
|
|
|
* Find the first bit set in a u_daddr_t.
|
2017-09-10 17:46:03 +00:00
|
|
|
*/
|
2019-05-10 22:49:01 +00:00
|
|
|
static inline int
|
2019-05-11 09:09:10 +00:00
|
|
|
generic_bitpos(u_daddr_t mask)
|
2019-05-10 22:49:01 +00:00
|
|
|
{
|
2019-05-11 02:13:52 +00:00
|
|
|
int hi, lo, mid;
|
2019-05-10 22:49:01 +00:00
|
|
|
|
2019-05-11 09:09:10 +00:00
|
|
|
lo = 0;
|
2020-07-25 18:29:10 +00:00
|
|
|
hi = BLIST_RADIX;
|
2019-05-11 09:09:10 +00:00
|
|
|
while (lo + 1 < hi) {
|
|
|
|
|
mid = (lo + hi) >> 1;
|
|
|
|
|
if (mask & bitrange(0, mid))
|
|
|
|
|
hi = mid;
|
|
|
|
|
else
|
|
|
|
|
lo = mid;
|
|
|
|
|
}
|
|
|
|
|
return (lo);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline int
|
|
|
|
|
bitpos(u_daddr_t mask)
|
|
|
|
|
{
|
|
|
|
|
|
2019-05-11 02:13:52 +00:00
|
|
|
switch (sizeof(mask)) {
|
2017-09-10 17:46:03 +00:00
|
|
|
#ifdef HAVE_INLINE_FFSLL
|
2019-05-11 02:13:52 +00:00
|
|
|
case sizeof(long long):
|
|
|
|
|
return (ffsll(mask) - 1);
|
2019-05-11 09:09:10 +00:00
|
|
|
#endif
|
|
|
|
|
#ifdef HAVE_INLINE_FFS
|
|
|
|
|
case sizeof(int):
|
|
|
|
|
return (ffs(mask) - 1);
|
2019-05-10 22:49:01 +00:00
|
|
|
#endif
|
2019-05-11 02:13:52 +00:00
|
|
|
default:
|
2019-05-11 09:09:10 +00:00
|
|
|
return (generic_bitpos(mask));
|
2019-05-11 02:13:52 +00:00
|
|
|
}
|
2017-08-01 03:51:26 +00:00
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* blist_create() - create a blist capable of handling up to the specified
|
|
|
|
|
* number of blocks
|
|
|
|
|
*
|
2011-12-03 15:41:37 +00:00
|
|
|
* blocks - must be greater than 0
|
2008-05-05 19:48:54 +00:00
|
|
|
* flags - malloc flags
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
2017-07-17 23:16:33 +00:00
|
|
|
* The smallest blist consists of a single leaf node capable of
|
2020-07-25 18:29:10 +00:00
|
|
|
* managing BLIST_RADIX blocks.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
blist_t
|
2008-05-05 19:48:54 +00:00
|
|
|
blist_create(daddr_t blocks, int flags)
|
1999-01-21 08:11:06 +00:00
|
|
|
{
|
|
|
|
|
blist_t bl;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
u_daddr_t nodes, radix;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2019-05-10 18:22:40 +00:00
|
|
|
KASSERT(blocks > 0, ("invalid block count"));
|
2018-09-05 19:05:30 +00:00
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
2018-09-05 19:05:30 +00:00
|
|
|
* Calculate the radix and node count used for scanning.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
nodes = 1;
|
2021-07-13 21:47:27 +00:00
|
|
|
for (radix = 1; (blocks - 1) / BLIST_RADIX / radix > 0;
|
|
|
|
|
radix *= BLIST_RADIX)
|
|
|
|
|
nodes += 1 + (blocks - 1) / BLIST_RADIX / radix;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Include a sentinel node to ensure that cross-leaf scans stay within
|
|
|
|
|
* the bounds of the allocation.
|
|
|
|
|
*/
|
|
|
|
|
if (blocks % BLIST_RADIX == 0)
|
|
|
|
|
nodes++;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-10-09 18:19:06 +00:00
|
|
|
bl = malloc(offsetof(struct blist, bl_root[nodes]), M_SWAP, flags |
|
|
|
|
|
M_ZERO);
|
2017-06-10 16:11:39 +00:00
|
|
|
if (bl == NULL)
|
|
|
|
|
return (NULL);
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
bl->bl_blocks = blocks;
|
|
|
|
|
bl->bl_radix = radix;
|
|
|
|
|
|
|
|
|
|
#if defined(BLIST_DEBUG)
|
|
|
|
|
printf(
|
2002-12-15 19:17:57 +00:00
|
|
|
"BLIST representing %lld blocks (%lld MB of swap)"
|
|
|
|
|
", requiring %lldK of ram\n",
|
|
|
|
|
(long long)bl->bl_blocks,
|
|
|
|
|
(long long)bl->bl_blocks * 4 / 1024,
|
2017-06-10 16:11:39 +00:00
|
|
|
(long long)(nodes * sizeof(blmeta_t) + 1023) / 1024
|
1999-01-21 08:11:06 +00:00
|
|
|
);
|
2002-12-15 19:17:57 +00:00
|
|
|
printf("BLIST raw radix tree contains %lld records\n",
|
2017-06-10 16:11:39 +00:00
|
|
|
(long long)nodes);
|
1999-01-21 08:11:06 +00:00
|
|
|
#endif
|
|
|
|
|
|
2017-06-10 16:11:39 +00:00
|
|
|
return (bl);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
2017-07-17 23:16:33 +00:00
|
|
|
void
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_destroy(blist_t bl)
|
|
|
|
|
{
|
The blst_radix_init function has two purposes - to compute the number of
nodes to allocate for the blist, and to initialize them. The computation
can be done much more quickly by identifying the terminating node, if any,
at every level of the tree and then summing the number of nodes at each
level that precedes the topmost terminator. The initialization can also be
done quickly, since settings at the root mark the tree as all-allocated, and
only a few terminator nodes need to be marked in the rest of the tree.
Eliminate blst_radix_init, and perform its two functions more simply in
blist_create.
The allocation of the blist takes places in two pieces, but there's no good
reason to do so, when a single allocation is sufficient, and simpler.
Allocate the blist struct, and the array of nodes associated with it, with a
single allocation.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11968
2017-10-08 22:17:39 +00:00
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
free(bl, M_SWAP);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
2017-06-13 17:49:49 +00:00
|
|
|
* blist_alloc() - reserve space in the block bitmap. Return the base
|
1999-01-21 08:11:06 +00:00
|
|
|
* of a contiguous region or SWAPBLK_NONE if space could
|
|
|
|
|
* not be allocated.
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
daddr_t
|
2019-05-11 16:15:13 +00:00
|
|
|
blist_alloc(blist_t bl, int *count, int maxcount)
|
1999-01-21 08:11:06 +00:00
|
|
|
{
|
2019-05-06 22:12:15 +00:00
|
|
|
daddr_t blk, cursor;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2019-05-11 16:15:13 +00:00
|
|
|
KASSERT(*count <= maxcount,
|
|
|
|
|
("invalid parameters %d > %d", *count, maxcount));
|
2019-07-06 06:15:03 +00:00
|
|
|
KASSERT(*count <= BLIST_MAX_ALLOC,
|
|
|
|
|
("minimum allocation too large: %d", *count));
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
|
2017-06-18 18:23:39 +00:00
|
|
|
/*
|
|
|
|
|
* This loop iterates at most twice. An allocation failure in the
|
|
|
|
|
* first iteration leads to a second iteration only if the cursor was
|
|
|
|
|
* non-zero. When the cursor is zero, an allocation failure will
|
2018-12-05 18:26:40 +00:00
|
|
|
* stop further iterations.
|
2017-06-18 18:23:39 +00:00
|
|
|
*/
|
2019-05-11 16:15:13 +00:00
|
|
|
for (cursor = bl->bl_cursor;; cursor = 0) {
|
|
|
|
|
blk = blst_meta_alloc(bl->bl_root, cursor, count, maxcount,
|
2017-08-13 16:39:49 +00:00
|
|
|
bl->bl_radix);
|
2017-06-18 18:23:39 +00:00
|
|
|
if (blk != SWAPBLK_NONE) {
|
2019-05-11 16:15:13 +00:00
|
|
|
bl->bl_avail -= *count;
|
|
|
|
|
bl->bl_cursor = blk + *count;
|
2017-08-25 18:47:23 +00:00
|
|
|
if (bl->bl_cursor == bl->bl_blocks)
|
|
|
|
|
bl->bl_cursor = 0;
|
2017-06-18 18:23:39 +00:00
|
|
|
return (blk);
|
2019-05-11 16:15:13 +00:00
|
|
|
}
|
|
|
|
|
if (cursor == 0)
|
2018-12-05 18:26:40 +00:00
|
|
|
return (SWAPBLK_NONE);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
2017-06-13 17:49:49 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_avail() - return the number of free blocks.
|
|
|
|
|
*/
|
|
|
|
|
daddr_t
|
|
|
|
|
blist_avail(blist_t bl)
|
|
|
|
|
{
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
return (bl->bl_avail);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_free() - free up space in the block bitmap. Return the base
|
2019-05-10 18:22:40 +00:00
|
|
|
* of a contiguous region.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
void
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(blist_t bl, daddr_t blkno, daddr_t count)
|
|
|
|
|
{
|
2017-07-24 17:23:53 +00:00
|
|
|
|
2019-05-10 18:22:40 +00:00
|
|
|
KASSERT(blkno >= 0 && blkno + count <= bl->bl_blocks,
|
|
|
|
|
("freeing invalid range: blkno %jx, count %d, blocks %jd",
|
|
|
|
|
(uintmax_t)blkno, (int)count, (uintmax_t)bl->bl_blocks));
|
2017-08-13 16:39:49 +00:00
|
|
|
blst_meta_free(bl->bl_root, blkno, count, bl->bl_radix);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
bl->bl_avail += count;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
2002-12-15 19:17:57 +00:00
|
|
|
/*
|
|
|
|
|
* blist_fill() - mark a region in the block bitmap as off-limits
|
|
|
|
|
* to the allocator (i.e. allocate it), ignoring any
|
|
|
|
|
* existing allocations. Return the number of blocks
|
|
|
|
|
* actually filled that were free before the call.
|
|
|
|
|
*/
|
2017-06-09 16:19:24 +00:00
|
|
|
daddr_t
|
2002-12-15 19:17:57 +00:00
|
|
|
blist_fill(blist_t bl, daddr_t blkno, daddr_t count)
|
|
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t filled;
|
2002-12-15 19:17:57 +00:00
|
|
|
|
2019-05-10 18:22:40 +00:00
|
|
|
KASSERT(blkno >= 0 && blkno + count <= bl->bl_blocks,
|
|
|
|
|
("filling invalid range: blkno %jx, count %d, blocks %jd",
|
|
|
|
|
(uintmax_t)blkno, (int)count, (uintmax_t)bl->bl_blocks));
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
filled = blst_meta_fill(bl->bl_root, blkno, count, bl->bl_radix);
|
|
|
|
|
bl->bl_avail -= filled;
|
|
|
|
|
return (filled);
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* blist_resize() - resize an existing radix tree to handle the
|
|
|
|
|
* specified number of blocks. This will reallocate
|
|
|
|
|
* the tree and transfer the previous bitmap to the new
|
|
|
|
|
* one. When extending the tree you can specify whether
|
|
|
|
|
* the new blocks are to left allocated or freed.
|
|
|
|
|
*/
|
|
|
|
|
void
|
2008-05-05 19:48:54 +00:00
|
|
|
blist_resize(blist_t *pbl, daddr_t count, int freenew, int flags)
|
1999-01-21 08:11:06 +00:00
|
|
|
{
|
2008-05-05 19:48:54 +00:00
|
|
|
blist_t newbl = blist_create(count, flags);
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_t save = *pbl;
|
|
|
|
|
|
|
|
|
|
*pbl = newbl;
|
|
|
|
|
if (count > save->bl_blocks)
|
|
|
|
|
count = save->bl_blocks;
|
2017-08-01 03:51:26 +00:00
|
|
|
blst_copy(save->bl_root, 0, save->bl_radix, newbl, count);
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If resizing upwards, should we free the new space or not?
|
|
|
|
|
*/
|
|
|
|
|
if (freenew && count < newbl->bl_blocks) {
|
|
|
|
|
blist_free(newbl, count, newbl->bl_blocks - count);
|
|
|
|
|
}
|
|
|
|
|
blist_destroy(save);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#ifdef BLIST_DEBUG
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_print() - dump radix tree
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
blist_print(blist_t bl)
|
|
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
printf("BLIST avail = %jd, cursor = %08jx {\n",
|
|
|
|
|
(uintmax_t)bl->bl_avail, (uintmax_t)bl->bl_cursor);
|
|
|
|
|
|
|
|
|
|
if (bl->bl_root->bm_bitmap != 0)
|
|
|
|
|
blst_radix_print(bl->bl_root, 0, bl->bl_radix, 4);
|
1999-01-21 08:11:06 +00:00
|
|
|
printf("}\n");
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
2017-09-10 17:46:03 +00:00
|
|
|
static const u_daddr_t fib[] = {
|
|
|
|
|
1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584,
|
|
|
|
|
4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811,
|
|
|
|
|
514229, 832040, 1346269, 2178309, 3524578,
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Use 'gap' to describe a maximal range of unallocated blocks/bits.
|
|
|
|
|
*/
|
|
|
|
|
struct gap_stats {
|
|
|
|
|
daddr_t start; /* current gap start, or SWAPBLK_NONE */
|
|
|
|
|
daddr_t num; /* number of gaps observed */
|
|
|
|
|
daddr_t max; /* largest gap size */
|
|
|
|
|
daddr_t avg; /* average gap size */
|
|
|
|
|
daddr_t err; /* sum - num * avg */
|
|
|
|
|
daddr_t histo[nitems(fib)]; /* # gaps in each size range */
|
|
|
|
|
int max_bucket; /* last histo elt with nonzero val */
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* gap_stats_counting() - is the state 'counting 1 bits'?
|
|
|
|
|
* or 'skipping 0 bits'?
|
|
|
|
|
*/
|
|
|
|
|
static inline bool
|
|
|
|
|
gap_stats_counting(const struct gap_stats *stats)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (stats->start != SWAPBLK_NONE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* init_gap_stats() - initialize stats on gap sizes
|
|
|
|
|
*/
|
|
|
|
|
static inline void
|
|
|
|
|
init_gap_stats(struct gap_stats *stats)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
bzero(stats, sizeof(*stats));
|
|
|
|
|
stats->start = SWAPBLK_NONE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* update_gap_stats() - update stats on gap sizes
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
update_gap_stats(struct gap_stats *stats, daddr_t posn)
|
|
|
|
|
{
|
|
|
|
|
daddr_t size;
|
|
|
|
|
int hi, lo, mid;
|
|
|
|
|
|
|
|
|
|
if (!gap_stats_counting(stats)) {
|
|
|
|
|
stats->start = posn;
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
size = posn - stats->start;
|
|
|
|
|
stats->start = SWAPBLK_NONE;
|
|
|
|
|
if (size > stats->max)
|
|
|
|
|
stats->max = size;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Find the fibonacci range that contains size,
|
|
|
|
|
* expecting to find it in an early range.
|
|
|
|
|
*/
|
|
|
|
|
lo = 0;
|
|
|
|
|
hi = 1;
|
|
|
|
|
while (hi < nitems(fib) && fib[hi] <= size) {
|
|
|
|
|
lo = hi;
|
|
|
|
|
hi *= 2;
|
|
|
|
|
}
|
|
|
|
|
if (hi >= nitems(fib))
|
|
|
|
|
hi = nitems(fib);
|
|
|
|
|
while (lo + 1 != hi) {
|
|
|
|
|
mid = (lo + hi) >> 1;
|
|
|
|
|
if (fib[mid] <= size)
|
|
|
|
|
lo = mid;
|
|
|
|
|
else
|
|
|
|
|
hi = mid;
|
|
|
|
|
}
|
|
|
|
|
stats->histo[lo]++;
|
|
|
|
|
if (lo > stats->max_bucket)
|
|
|
|
|
stats->max_bucket = lo;
|
|
|
|
|
stats->err += size - stats->avg;
|
|
|
|
|
stats->num++;
|
|
|
|
|
stats->avg += stats->err / stats->num;
|
|
|
|
|
stats->err %= stats->num;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* dump_gap_stats() - print stats on gap sizes
|
|
|
|
|
*/
|
|
|
|
|
static inline void
|
|
|
|
|
dump_gap_stats(const struct gap_stats *stats, struct sbuf *s)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
sbuf_printf(s, "number of maximal free ranges: %jd\n",
|
|
|
|
|
(intmax_t)stats->num);
|
|
|
|
|
sbuf_printf(s, "largest free range: %jd\n", (intmax_t)stats->max);
|
|
|
|
|
sbuf_printf(s, "average maximal free range size: %jd\n",
|
|
|
|
|
(intmax_t)stats->avg);
|
|
|
|
|
sbuf_printf(s, "number of maximal free ranges of different sizes:\n");
|
|
|
|
|
sbuf_printf(s, " count | size range\n");
|
|
|
|
|
sbuf_printf(s, " ----- | ----------\n");
|
|
|
|
|
for (i = 0; i < stats->max_bucket; i++) {
|
|
|
|
|
if (stats->histo[i] != 0) {
|
|
|
|
|
sbuf_printf(s, "%20jd | ",
|
|
|
|
|
(intmax_t)stats->histo[i]);
|
|
|
|
|
if (fib[i] != fib[i + 1] - 1)
|
|
|
|
|
sbuf_printf(s, "%jd to %jd\n", (intmax_t)fib[i],
|
|
|
|
|
(intmax_t)fib[i + 1] - 1);
|
|
|
|
|
else
|
|
|
|
|
sbuf_printf(s, "%jd\n", (intmax_t)fib[i]);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
sbuf_printf(s, "%20jd | ", (intmax_t)stats->histo[i]);
|
|
|
|
|
if (stats->histo[i] > 1)
|
|
|
|
|
sbuf_printf(s, "%jd to %jd\n", (intmax_t)fib[i],
|
|
|
|
|
(intmax_t)stats->max);
|
|
|
|
|
else
|
|
|
|
|
sbuf_printf(s, "%jd\n", (intmax_t)stats->max);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_stats() - dump radix tree stats
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
blist_stats(blist_t bl, struct sbuf *s)
|
|
|
|
|
{
|
|
|
|
|
struct gap_stats gstats;
|
|
|
|
|
struct gap_stats *stats = &gstats;
|
|
|
|
|
daddr_t i, nodes, radix;
|
2019-05-11 09:09:10 +00:00
|
|
|
u_daddr_t diff, mask;
|
|
|
|
|
int digit;
|
2017-09-10 17:46:03 +00:00
|
|
|
|
|
|
|
|
init_gap_stats(stats);
|
|
|
|
|
nodes = 0;
|
2020-07-25 18:29:10 +00:00
|
|
|
radix = bl->bl_radix;
|
|
|
|
|
for (i = 0; i < bl->bl_blocks; ) {
|
2017-09-10 17:46:03 +00:00
|
|
|
/*
|
|
|
|
|
* Check for skippable subtrees starting at i.
|
|
|
|
|
*/
|
2020-07-25 18:29:10 +00:00
|
|
|
while (radix != 1) {
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (bl->bl_root[nodes].bm_bitmap == 0) {
|
2017-09-10 17:46:03 +00:00
|
|
|
if (gap_stats_counting(stats))
|
|
|
|
|
update_gap_stats(stats, i);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Skip subtree root.
|
|
|
|
|
*/
|
|
|
|
|
nodes++;
|
2020-07-25 18:29:10 +00:00
|
|
|
radix /= BLIST_RADIX;
|
2017-09-10 17:46:03 +00:00
|
|
|
}
|
2020-07-25 18:29:10 +00:00
|
|
|
if (radix == 1) {
|
2017-09-10 17:46:03 +00:00
|
|
|
/*
|
|
|
|
|
* Scan leaf.
|
|
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = bl->bl_root[nodes].bm_bitmap;
|
2017-09-10 17:46:03 +00:00
|
|
|
diff = mask ^ (mask << 1);
|
|
|
|
|
if (gap_stats_counting(stats))
|
|
|
|
|
diff ^= 1;
|
|
|
|
|
while (diff != 0) {
|
2019-05-11 09:09:10 +00:00
|
|
|
digit = bitpos(diff);
|
|
|
|
|
update_gap_stats(stats, i + digit);
|
|
|
|
|
diff ^= bitrange(digit, 1);
|
2017-09-10 17:46:03 +00:00
|
|
|
}
|
|
|
|
|
}
|
2020-07-25 18:29:10 +00:00
|
|
|
nodes += radix_to_skip(radix * BLIST_RADIX);
|
|
|
|
|
i += radix * BLIST_RADIX;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Find max size subtree starting at i.
|
|
|
|
|
*/
|
|
|
|
|
for (radix = 1;
|
|
|
|
|
((i / BLIST_RADIX / radix) & BLIST_MASK) == 0;
|
|
|
|
|
radix *= BLIST_RADIX)
|
|
|
|
|
;
|
2017-09-10 17:46:03 +00:00
|
|
|
}
|
|
|
|
|
update_gap_stats(stats, i);
|
|
|
|
|
dump_gap_stats(stats, s);
|
|
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/************************************************************************
|
|
|
|
|
* ALLOCATION SUPPORT FUNCTIONS *
|
|
|
|
|
************************************************************************
|
|
|
|
|
*
|
2017-07-17 23:16:33 +00:00
|
|
|
* These support functions do all the actual work. They may seem
|
1999-01-21 08:11:06 +00:00
|
|
|
* rather longish, but that's because I've commented them up. The
|
|
|
|
|
* actual code is straight forward.
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
2019-07-06 06:15:03 +00:00
|
|
|
* BLST_NEXT_LEAF_ALLOC() - allocate the blocks starting with the next leaf.
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
*
|
2019-07-06 06:15:03 +00:00
|
|
|
* 'scan' is a leaf node, and its first block is at address 'start'. The
|
|
|
|
|
* next leaf node could be adjacent, or several nodes away if the least
|
|
|
|
|
* common ancestor of 'scan' and its neighbor is several levels up. Use
|
|
|
|
|
* addresses to determine how many meta-nodes lie between the leaves. If
|
|
|
|
|
* sequence of leaves starting with the next one has enough initial bits
|
|
|
|
|
* set, clear them and clear the bits in the meta nodes on the path up to
|
|
|
|
|
* the least common ancestor to mark any subtrees made completely empty.
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
*/
|
|
|
|
|
static int
|
2019-07-06 06:15:03 +00:00
|
|
|
blst_next_leaf_alloc(blmeta_t *scan, daddr_t start, int count, int maxcount)
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
{
|
|
|
|
|
u_daddr_t radix;
|
2019-07-06 06:15:03 +00:00
|
|
|
daddr_t blk;
|
2019-05-11 16:15:13 +00:00
|
|
|
int avail, digit;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
start += BLIST_RADIX;
|
|
|
|
|
for (blk = start; blk - start < maxcount; blk += BLIST_RADIX) {
|
2019-07-06 06:15:03 +00:00
|
|
|
/* Skip meta-nodes, as long as they promise more free blocks. */
|
2020-07-25 18:29:10 +00:00
|
|
|
radix = BLIST_RADIX;
|
2019-07-06 06:15:03 +00:00
|
|
|
while (((++scan)->bm_bitmap & 1) == 1 &&
|
2020-07-25 18:29:10 +00:00
|
|
|
((blk / radix) & BLIST_MASK) == 0)
|
|
|
|
|
radix *= BLIST_RADIX;
|
2019-07-06 06:15:03 +00:00
|
|
|
if (~scan->bm_bitmap != 0) {
|
|
|
|
|
/*
|
|
|
|
|
* Either there is no next leaf with any free blocks,
|
|
|
|
|
* or we've reached the next leaf and found that some
|
|
|
|
|
* of its blocks are not free. In the first case,
|
|
|
|
|
* bitpos() returns zero here.
|
|
|
|
|
*/
|
|
|
|
|
avail = blk - start + bitpos(~scan->bm_bitmap);
|
2019-07-11 20:52:39 +00:00
|
|
|
if (avail < count || avail == 0) {
|
2019-07-06 06:15:03 +00:00
|
|
|
/*
|
|
|
|
|
* There isn't a next leaf with enough free
|
2019-07-11 20:52:39 +00:00
|
|
|
* blocks at its beginning to bother
|
|
|
|
|
* allocating.
|
2019-07-06 06:15:03 +00:00
|
|
|
*/
|
|
|
|
|
return (avail);
|
|
|
|
|
}
|
|
|
|
|
maxcount = imin(avail, maxcount);
|
2020-07-25 18:29:10 +00:00
|
|
|
if (maxcount % BLIST_RADIX == 0) {
|
2019-07-11 20:52:39 +00:00
|
|
|
/*
|
|
|
|
|
* There was no next leaf. Back scan up to
|
|
|
|
|
* last leaf.
|
|
|
|
|
*/
|
2020-07-25 18:29:10 +00:00
|
|
|
do {
|
|
|
|
|
radix /= BLIST_RADIX;
|
2019-07-11 20:52:39 +00:00
|
|
|
--scan;
|
2020-07-25 18:29:10 +00:00
|
|
|
} while (radix != 1);
|
|
|
|
|
blk -= BLIST_RADIX;
|
2019-07-11 20:52:39 +00:00
|
|
|
}
|
2019-07-06 06:15:03 +00:00
|
|
|
}
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
}
|
2020-09-01 22:12:32 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
/*
|
2019-07-06 06:15:03 +00:00
|
|
|
* 'scan' is the last leaf that provides blocks. Clear from 1 to
|
2020-07-25 18:29:10 +00:00
|
|
|
* BLIST_RADIX bits to represent the allocation of those last blocks.
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
*/
|
2020-07-25 18:29:10 +00:00
|
|
|
if (maxcount % BLIST_RADIX != 0)
|
|
|
|
|
scan->bm_bitmap &= ~bitrange(0, maxcount % BLIST_RADIX);
|
2019-07-06 06:15:03 +00:00
|
|
|
else
|
|
|
|
|
scan->bm_bitmap = 0;
|
|
|
|
|
|
|
|
|
|
for (;;) {
|
|
|
|
|
/* Back up over meta-nodes, clearing bits if necessary. */
|
2020-07-25 18:29:10 +00:00
|
|
|
blk -= BLIST_RADIX;
|
|
|
|
|
for (radix = BLIST_RADIX;
|
|
|
|
|
(digit = ((blk / radix) & BLIST_MASK)) == 0;
|
|
|
|
|
radix *= BLIST_RADIX) {
|
2019-07-06 06:15:03 +00:00
|
|
|
if ((scan--)->bm_bitmap == 0)
|
|
|
|
|
scan->bm_bitmap ^= 1;
|
|
|
|
|
}
|
|
|
|
|
if ((scan--)->bm_bitmap == 0)
|
2019-05-10 18:25:06 +00:00
|
|
|
scan[-digit * radix_to_skip(radix)].bm_bitmap ^=
|
|
|
|
|
(u_daddr_t)1 << digit;
|
2019-07-06 06:15:03 +00:00
|
|
|
|
|
|
|
|
if (blk == start)
|
2019-05-10 18:25:06 +00:00
|
|
|
break;
|
2019-07-06 06:15:03 +00:00
|
|
|
/* Clear all the bits of this leaf. */
|
|
|
|
|
scan->bm_bitmap = 0;
|
|
|
|
|
}
|
|
|
|
|
return (maxcount);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLST_LEAF_ALLOC() - allocate at a leaf in the radix tree (a bitmap).
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
blst_leaf_alloc updates bighint for a leaf when an allocation is successful
and includes the last block represented by the leaf. The reasoning is that,
if the last block is included, then there must be no solution before that
one in the leaf, so the leaf cannot provide an allocation that big again;
indeed, the leaf cannot provide a solution bigger than range1.
Which is all correct, except that if the value of blk passed in did not
represent the first block of the leaf, because the cursor was pointing to
the middle of the leaf, then a possible solution before the cursor may have
been ignored, and bighint cannot be updated.
Consider the sequence allocate 63 (returning address 0), free 0,63 (freeing
that same block, and allocate 1 (returning 63). The result is that one
block is allocated from the first leaf, and the value of bighint is 0, so
that nothing can be allocated from that leaf until the only block allocated
from that leaf is freed. This change detects that skipped-over solution,
and when there is one it makes sure that the value of bighint is not changed
when the last block is allocated.
Submitted by: Doug Moore <dougm@rice.edu>
Tested by: pho
X-MFC with: r340402
Differential Revision: https://reviews.freebsd.org/D18474
2018-12-09 17:55:10 +00:00
|
|
|
* This function is the core of the allocator. Its execution time is
|
|
|
|
|
* proportional to log(count), plus height of the tree if the allocation
|
|
|
|
|
* crosses a leaf boundary.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
|
|
|
|
static daddr_t
|
2019-05-11 16:15:13 +00:00
|
|
|
blst_leaf_alloc(blmeta_t *scan, daddr_t blk, int *count, int maxcount)
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
{
|
2019-12-14 19:44:42 +00:00
|
|
|
u_daddr_t mask;
|
|
|
|
|
int bighint, count1, hi, lo, num_shifts;
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
|
2019-05-11 16:15:13 +00:00
|
|
|
count1 = *count - 1;
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
num_shifts = fls(count1);
|
2019-12-14 19:44:42 +00:00
|
|
|
mask = ~scan->bm_bitmap;
|
|
|
|
|
while ((mask & (mask + 1)) != 0 && num_shifts > 0) {
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
2019-12-14 19:44:42 +00:00
|
|
|
* If bit i is 0 in mask, then bits in [i, i + (count1 >>
|
|
|
|
|
* num_shifts)] are 1 in scan->bm_bitmap. Reduce num_shifts to
|
|
|
|
|
* 0, while preserving this invariant. The updates to mask
|
|
|
|
|
* leave fewer bits 0, but each bit that remains 0 represents a
|
|
|
|
|
* longer string of consecutive 1-bits in scan->bm_bitmap. If
|
|
|
|
|
* more updates to mask cannot set more bits, because mask is
|
|
|
|
|
* partitioned with all 1 bits following all 0 bits, the loop
|
|
|
|
|
* terminates immediately.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
num_shifts--;
|
2019-12-14 19:44:42 +00:00
|
|
|
mask |= mask >> ((count1 >> num_shifts) + 1) / 2;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
2019-12-14 19:44:42 +00:00
|
|
|
bighint = count1 >> num_shifts;
|
|
|
|
|
if (~mask == 0) {
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
2019-12-14 19:44:42 +00:00
|
|
|
* Update bighint. There is no allocation bigger than
|
|
|
|
|
* count1 >> num_shifts starting in this leaf.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
2019-12-14 19:44:42 +00:00
|
|
|
scan->bm_bighint = bighint;
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
return (SWAPBLK_NONE);
|
|
|
|
|
}
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
/* Discard any candidates that appear before blk. */
|
2020-07-25 18:29:10 +00:00
|
|
|
if ((blk & BLIST_MASK) != 0) {
|
|
|
|
|
if ((~mask & bitrange(0, blk & BLIST_MASK)) != 0) {
|
2019-12-14 19:44:42 +00:00
|
|
|
/* Grow bighint in case all discarded bits are set. */
|
2020-07-25 18:29:10 +00:00
|
|
|
bighint += blk & BLIST_MASK;
|
|
|
|
|
mask |= bitrange(0, blk & BLIST_MASK);
|
2019-12-14 19:44:42 +00:00
|
|
|
if (~mask == 0) {
|
|
|
|
|
scan->bm_bighint = bighint;
|
blst_leaf_alloc updates bighint for a leaf when an allocation is successful
and includes the last block represented by the leaf. The reasoning is that,
if the last block is included, then there must be no solution before that
one in the leaf, so the leaf cannot provide an allocation that big again;
indeed, the leaf cannot provide a solution bigger than range1.
Which is all correct, except that if the value of blk passed in did not
represent the first block of the leaf, because the cursor was pointing to
the middle of the leaf, then a possible solution before the cursor may have
been ignored, and bighint cannot be updated.
Consider the sequence allocate 63 (returning address 0), free 0,63 (freeing
that same block, and allocate 1 (returning 63). The result is that one
block is allocated from the first leaf, and the value of bighint is 0, so
that nothing can be allocated from that leaf until the only block allocated
from that leaf is freed. This change detects that skipped-over solution,
and when there is one it makes sure that the value of bighint is not changed
when the last block is allocated.
Submitted by: Doug Moore <dougm@rice.edu>
Tested by: pho
X-MFC with: r340402
Differential Revision: https://reviews.freebsd.org/D18474
2018-12-09 17:55:10 +00:00
|
|
|
return (SWAPBLK_NONE);
|
2019-12-14 19:44:42 +00:00
|
|
|
}
|
blst_leaf_alloc updates bighint for a leaf when an allocation is successful
and includes the last block represented by the leaf. The reasoning is that,
if the last block is included, then there must be no solution before that
one in the leaf, so the leaf cannot provide an allocation that big again;
indeed, the leaf cannot provide a solution bigger than range1.
Which is all correct, except that if the value of blk passed in did not
represent the first block of the leaf, because the cursor was pointing to
the middle of the leaf, then a possible solution before the cursor may have
been ignored, and bighint cannot be updated.
Consider the sequence allocate 63 (returning address 0), free 0,63 (freeing
that same block, and allocate 1 (returning 63). The result is that one
block is allocated from the first leaf, and the value of bighint is 0, so
that nothing can be allocated from that leaf until the only block allocated
from that leaf is freed. This change detects that skipped-over solution,
and when there is one it makes sure that the value of bighint is not changed
when the last block is allocated.
Submitted by: Doug Moore <dougm@rice.edu>
Tested by: pho
X-MFC with: r340402
Differential Revision: https://reviews.freebsd.org/D18474
2018-12-09 17:55:10 +00:00
|
|
|
}
|
2020-07-25 18:29:10 +00:00
|
|
|
blk -= blk & BLIST_MASK;
|
blst_leaf_alloc updates bighint for a leaf when an allocation is successful
and includes the last block represented by the leaf. The reasoning is that,
if the last block is included, then there must be no solution before that
one in the leaf, so the leaf cannot provide an allocation that big again;
indeed, the leaf cannot provide a solution bigger than range1.
Which is all correct, except that if the value of blk passed in did not
represent the first block of the leaf, because the cursor was pointing to
the middle of the leaf, then a possible solution before the cursor may have
been ignored, and bighint cannot be updated.
Consider the sequence allocate 63 (returning address 0), free 0,63 (freeing
that same block, and allocate 1 (returning 63). The result is that one
block is allocated from the first leaf, and the value of bighint is 0, so
that nothing can be allocated from that leaf until the only block allocated
from that leaf is freed. This change detects that skipped-over solution,
and when there is one it makes sure that the value of bighint is not changed
when the last block is allocated.
Submitted by: Doug Moore <dougm@rice.edu>
Tested by: pho
X-MFC with: r340402
Differential Revision: https://reviews.freebsd.org/D18474
2018-12-09 17:55:10 +00:00
|
|
|
}
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The least significant set bit in mask marks the start of the first
|
2019-05-11 16:15:13 +00:00
|
|
|
* available range of sufficient size. Find its position.
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
*/
|
2019-12-14 19:44:42 +00:00
|
|
|
lo = bitpos(~mask);
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
|
2019-05-11 16:15:13 +00:00
|
|
|
/*
|
|
|
|
|
* Find how much space is available starting at that position.
|
|
|
|
|
*/
|
2019-12-14 19:44:42 +00:00
|
|
|
if ((mask & (mask + 1)) != 0) {
|
2019-05-11 16:15:13 +00:00
|
|
|
/* Count the 1 bits starting at position lo. */
|
2019-12-14 19:44:42 +00:00
|
|
|
hi = bitpos(mask & (mask + 1)) + count1;
|
2019-05-11 16:15:13 +00:00
|
|
|
if (maxcount < hi - lo)
|
|
|
|
|
hi = lo + maxcount;
|
|
|
|
|
*count = hi - lo;
|
2019-12-14 19:44:42 +00:00
|
|
|
mask = ~bitrange(lo, *count);
|
2020-07-25 18:29:10 +00:00
|
|
|
} else if (maxcount <= BLIST_RADIX - lo) {
|
2019-05-11 16:15:13 +00:00
|
|
|
/* All the blocks we can use are available here. */
|
|
|
|
|
hi = lo + maxcount;
|
|
|
|
|
*count = maxcount;
|
2019-12-14 19:44:42 +00:00
|
|
|
mask = ~bitrange(lo, *count);
|
2020-07-25 18:29:10 +00:00
|
|
|
if (hi == BLIST_RADIX)
|
2019-12-14 19:44:42 +00:00
|
|
|
scan->bm_bighint = bighint;
|
2019-05-11 16:15:13 +00:00
|
|
|
} else {
|
|
|
|
|
/* Check next leaf for some of the blocks we want or need. */
|
2020-07-25 18:29:10 +00:00
|
|
|
count1 = *count - (BLIST_RADIX - lo);
|
|
|
|
|
maxcount -= BLIST_RADIX - lo;
|
2019-05-11 16:15:13 +00:00
|
|
|
hi = blst_next_leaf_alloc(scan, blk, count1, maxcount);
|
|
|
|
|
if (hi < count1)
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
/*
|
2019-05-11 16:15:13 +00:00
|
|
|
* The next leaf cannot supply enough blocks to reach
|
|
|
|
|
* the minimum required allocation. The hint cannot be
|
|
|
|
|
* updated, because the same allocation request could
|
|
|
|
|
* be satisfied later, by this leaf, if the state of
|
|
|
|
|
* the next leaf changes, and without any changes to
|
|
|
|
|
* this leaf.
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
*/
|
|
|
|
|
return (SWAPBLK_NONE);
|
2020-07-25 18:29:10 +00:00
|
|
|
*count = BLIST_RADIX - lo + hi;
|
2019-12-14 19:44:42 +00:00
|
|
|
scan->bm_bighint = bighint;
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Clear the allocated bits from this leaf. */
|
2019-12-14 19:44:42 +00:00
|
|
|
scan->bm_bitmap &= mask;
|
blst_leaf_alloc updates bighint for a leaf when an allocation is successful
and includes the last block represented by the leaf. The reasoning is that,
if the last block is included, then there must be no solution before that
one in the leaf, so the leaf cannot provide an allocation that big again;
indeed, the leaf cannot provide a solution bigger than range1.
Which is all correct, except that if the value of blk passed in did not
represent the first block of the leaf, because the cursor was pointing to
the middle of the leaf, then a possible solution before the cursor may have
been ignored, and bighint cannot be updated.
Consider the sequence allocate 63 (returning address 0), free 0,63 (freeing
that same block, and allocate 1 (returning 63). The result is that one
block is allocated from the first leaf, and the value of bighint is 0, so
that nothing can be allocated from that leaf until the only block allocated
from that leaf is freed. This change detects that skipped-over solution,
and when there is one it makes sure that the value of bighint is not changed
when the last block is allocated.
Submitted by: Doug Moore <dougm@rice.edu>
Tested by: pho
X-MFC with: r340402
Differential Revision: https://reviews.freebsd.org/D18474
2018-12-09 17:55:10 +00:00
|
|
|
return (blk + lo);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_meta_alloc() - allocate at a meta in the radix tree.
|
|
|
|
|
*
|
|
|
|
|
* Attempt to allocate at a meta node. If we can't, we update
|
|
|
|
|
* bighint and return a failure. Updating bighint optimize future
|
|
|
|
|
* calls that hit this node. We have to check for our collapse cases
|
|
|
|
|
* and we have a few optimizations strewn in as well.
|
|
|
|
|
*/
|
|
|
|
|
static daddr_t
|
2019-05-11 16:15:13 +00:00
|
|
|
blst_meta_alloc(blmeta_t *scan, daddr_t cursor, int *count,
|
|
|
|
|
int maxcount, u_daddr_t radix)
|
2017-06-18 18:23:39 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t blk, i, r, skip;
|
2019-05-11 09:09:10 +00:00
|
|
|
u_daddr_t mask;
|
2017-06-18 18:23:39 +00:00
|
|
|
bool scan_from_start;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
int digit;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
if (radix == 1)
|
2019-05-11 16:15:13 +00:00
|
|
|
return (blst_leaf_alloc(scan, cursor, count, maxcount));
|
2020-07-25 18:29:10 +00:00
|
|
|
blk = cursor & -(radix * BLIST_RADIX);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan_from_start = (cursor == blk);
|
2017-08-01 03:51:26 +00:00
|
|
|
skip = radix_to_skip(radix);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = scan->bm_bitmap;
|
|
|
|
|
|
|
|
|
|
/* Discard any candidates that appear before cursor. */
|
2020-07-25 18:29:10 +00:00
|
|
|
digit = (cursor / radix) & BLIST_MASK;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask &= (u_daddr_t)-1 << digit;
|
2018-11-24 21:52:10 +00:00
|
|
|
if (mask == 0)
|
|
|
|
|
return (SWAPBLK_NONE);
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* If the first try is for a block that includes the cursor, pre-undo
|
|
|
|
|
* the digit * radix offset in the first call; otherwise, ignore the
|
|
|
|
|
* cursor entirely.
|
2017-06-13 17:49:49 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (((mask >> digit) & 1) == 1)
|
|
|
|
|
cursor -= digit * radix;
|
|
|
|
|
else
|
|
|
|
|
cursor = blk;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
/*
|
|
|
|
|
* Examine the nonempty subtree associated with each bit set in mask.
|
|
|
|
|
*/
|
|
|
|
|
do {
|
2019-05-11 09:09:10 +00:00
|
|
|
digit = bitpos(mask);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
i = 1 + digit * skip;
|
2019-05-11 16:15:13 +00:00
|
|
|
if (*count <= scan[i].bm_bighint) {
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* The allocation might fit beginning in the i'th subtree.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
r = blst_meta_alloc(&scan[i], cursor + digit * radix,
|
2020-07-25 18:29:10 +00:00
|
|
|
count, maxcount, radix / BLIST_RADIX);
|
1999-01-21 08:11:06 +00:00
|
|
|
if (r != SWAPBLK_NONE) {
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (scan[i].bm_bitmap == 0)
|
2019-05-11 09:09:10 +00:00
|
|
|
scan->bm_bitmap ^= bitrange(digit, 1);
|
2017-06-13 17:49:49 +00:00
|
|
|
return (r);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
}
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
cursor = blk;
|
2019-05-11 09:09:10 +00:00
|
|
|
} while ((mask ^= bitrange(digit, 1)) != 0);
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* We couldn't allocate count in this subtree. If the whole tree was
|
|
|
|
|
* scanned, and the last tree node is allocated, update bighint.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
2020-07-25 18:29:10 +00:00
|
|
|
if (scan_from_start && !(digit == BLIST_RADIX - 1 &&
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan[i].bm_bighint == BLIST_MAX_ALLOC))
|
2019-05-11 16:15:13 +00:00
|
|
|
scan->bm_bighint = *count - 1;
|
2017-06-18 18:23:39 +00:00
|
|
|
|
|
|
|
|
return (SWAPBLK_NONE);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLST_LEAF_FREE() - free allocated block from leaf bitmap
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
static void
|
2017-07-17 23:16:33 +00:00
|
|
|
blst_leaf_free(blmeta_t *scan, daddr_t blk, int count)
|
|
|
|
|
{
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
u_daddr_t mask;
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* free some data in this bitmap
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* mask=0000111111111110000
|
1999-01-21 08:11:06 +00:00
|
|
|
* \_________/\__/
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* count n
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
2020-07-25 18:29:10 +00:00
|
|
|
mask = bitrange(blk & BLIST_MASK, count);
|
2019-05-10 18:22:40 +00:00
|
|
|
KASSERT((scan->bm_bitmap & mask) == 0,
|
|
|
|
|
("freeing free block: %jx, size %d, mask %jx",
|
|
|
|
|
(uintmax_t)blk, count, (uintmax_t)scan->bm_bitmap & mask));
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bitmap |= mask;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLST_META_FREE() - free allocated blocks from radix tree meta info
|
|
|
|
|
*
|
|
|
|
|
* This support routine frees a range of blocks from the bitmap.
|
|
|
|
|
* The range must be entirely enclosed by this radix node. If a
|
|
|
|
|
* meta node, we break the range down recursively to free blocks
|
|
|
|
|
* in subnodes (which means that this code can free an arbitrary
|
|
|
|
|
* range whereas the allocation code cannot allocate an arbitrary
|
|
|
|
|
* range).
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
static void
|
2017-08-13 16:39:49 +00:00
|
|
|
blst_meta_free(blmeta_t *scan, daddr_t freeBlk, daddr_t count, u_daddr_t radix)
|
2017-06-27 17:45:26 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t blk, endBlk, i, skip;
|
|
|
|
|
int digit, endDigit;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* We could probably do a better job here. We are required to make
|
|
|
|
|
* bighint at least as large as the biggest allocable block of data.
|
|
|
|
|
* If we just shoehorn it, a little extra overhead will be incurred
|
|
|
|
|
* on the next allocation (but only that one typically).
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bighint = BLIST_MAX_ALLOC;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
if (radix == 1)
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
return (blst_leaf_free(scan, freeBlk, count));
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
endBlk = freeBlk + count;
|
|
|
|
|
blk = (freeBlk + radix * BLIST_RADIX) & -(radix * BLIST_RADIX);
|
|
|
|
|
/*
|
|
|
|
|
* blk is first block past the end of the range of this meta node,
|
|
|
|
|
* or 0 in case of overflow.
|
|
|
|
|
*/
|
|
|
|
|
if (blk != 0)
|
|
|
|
|
endBlk = ummin(endBlk, blk);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
skip = radix_to_skip(radix);
|
|
|
|
|
blk = freeBlk & -radix;
|
2020-07-25 18:29:10 +00:00
|
|
|
digit = (blk / radix) & BLIST_MASK;
|
|
|
|
|
endDigit = 1 + (((endBlk - 1) / radix) & BLIST_MASK);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bitmap |= bitrange(digit, endDigit - digit);
|
|
|
|
|
for (i = 1 + digit * skip; blk < endBlk; i += skip) {
|
1999-01-21 08:11:06 +00:00
|
|
|
blk += radix;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
count = ummin(blk, endBlk) - freeBlk;
|
2020-07-25 18:29:10 +00:00
|
|
|
blst_meta_free(&scan[i], freeBlk, count, radix / BLIST_RADIX);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
freeBlk = blk;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* BLST_COPY() - copy one radix tree to another
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
|
|
|
|
* Locates free space in the source tree and frees it in the destination
|
|
|
|
|
* tree. The space may not already be free in the destination.
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
static void
|
2017-08-01 03:51:26 +00:00
|
|
|
blst_copy(blmeta_t *scan, daddr_t blk, daddr_t radix, blist_t dest,
|
|
|
|
|
daddr_t count)
|
2017-07-17 23:16:33 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t endBlk, i, skip;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Leaf node
|
|
|
|
|
*/
|
|
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
if (radix == 1) {
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
u_daddr_t v = scan->bm_bitmap;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
if (v == (u_daddr_t)-1) {
|
|
|
|
|
blist_free(dest, blk, count);
|
|
|
|
|
} else if (v != 0) {
|
|
|
|
|
int i;
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
for (i = 0; i < count; ++i) {
|
2017-06-05 17:14:16 +00:00
|
|
|
if (v & ((u_daddr_t)1 << i))
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(dest, blk + i, 1);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Meta node
|
|
|
|
|
*/
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (scan->bm_bitmap == 0) {
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* Source all allocated, leave dest allocated
|
|
|
|
|
*/
|
|
|
|
|
return;
|
2017-07-17 23:16:33 +00:00
|
|
|
}
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
endBlk = blk + count;
|
|
|
|
|
skip = radix_to_skip(radix);
|
|
|
|
|
for (i = 1; blk < endBlk; i += skip) {
|
1999-01-21 08:11:06 +00:00
|
|
|
blk += radix;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
count = radix;
|
|
|
|
|
if (blk >= endBlk)
|
|
|
|
|
count -= blk - endBlk;
|
2020-07-25 18:29:10 +00:00
|
|
|
blst_copy(&scan[i], blk - radix,
|
|
|
|
|
radix / BLIST_RADIX, dest, count);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2002-12-15 19:17:57 +00:00
|
|
|
/*
|
|
|
|
|
* BLST_LEAF_FILL() - allocate specific blocks in leaf bitmap
|
|
|
|
|
*
|
|
|
|
|
* This routine allocates all blocks in the specified range
|
|
|
|
|
* regardless of any existing allocations in that range. Returns
|
|
|
|
|
* the number of blocks allocated by the call.
|
|
|
|
|
*/
|
2017-06-09 16:19:24 +00:00
|
|
|
static daddr_t
|
2002-12-15 19:17:57 +00:00
|
|
|
blst_leaf_fill(blmeta_t *scan, daddr_t blk, int count)
|
|
|
|
|
{
|
2017-06-09 16:19:24 +00:00
|
|
|
daddr_t nblks;
|
2017-06-13 17:49:49 +00:00
|
|
|
u_daddr_t mask;
|
2002-12-15 19:17:57 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
mask = bitrange(blk & BLIST_MASK, count);
|
2002-12-15 19:17:57 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
/* Count the number of blocks that we are allocating. */
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
nblks = bitcount64(scan->bm_bitmap & mask);
|
2002-12-15 19:17:57 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bitmap &= ~mask;
|
2017-06-13 17:49:49 +00:00
|
|
|
return (nblks);
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLIST_META_FILL() - allocate specific blocks at a meta node
|
|
|
|
|
*
|
|
|
|
|
* This routine allocates the specified range of blocks,
|
|
|
|
|
* regardless of any existing allocations in the range. The
|
|
|
|
|
* range must be within the extent of this node. Returns the
|
|
|
|
|
* number of blocks allocated by the call.
|
|
|
|
|
*/
|
2017-06-09 16:19:24 +00:00
|
|
|
static daddr_t
|
2017-08-13 16:39:49 +00:00
|
|
|
blst_meta_fill(blmeta_t *scan, daddr_t allocBlk, daddr_t count, u_daddr_t radix)
|
2017-06-27 17:45:26 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t blk, endBlk, i, nblks, skip;
|
|
|
|
|
int digit;
|
2002-12-15 19:17:57 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
if (radix == 1)
|
2017-07-24 17:23:53 +00:00
|
|
|
return (blst_leaf_fill(scan, allocBlk, count));
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
endBlk = allocBlk + count;
|
|
|
|
|
blk = (allocBlk + radix * BLIST_RADIX) & -(radix * BLIST_RADIX);
|
|
|
|
|
/*
|
|
|
|
|
* blk is first block past the end of the range of this meta node,
|
|
|
|
|
* or 0 in case of overflow.
|
|
|
|
|
*/
|
|
|
|
|
if (blk != 0)
|
|
|
|
|
endBlk = ummin(endBlk, blk);
|
2017-08-01 03:51:26 +00:00
|
|
|
skip = radix_to_skip(radix);
|
2017-08-13 16:39:49 +00:00
|
|
|
blk = allocBlk & -radix;
|
2017-06-27 17:45:26 +00:00
|
|
|
nblks = 0;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
while (blk < endBlk) {
|
2020-07-25 18:29:10 +00:00
|
|
|
digit = (blk / radix) & BLIST_MASK;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
i = 1 + digit * skip;
|
2002-12-15 19:17:57 +00:00
|
|
|
blk += radix;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
count = ummin(blk, endBlk) - allocBlk;
|
2020-07-25 18:29:10 +00:00
|
|
|
nblks += blst_meta_fill(&scan[i], allocBlk, count,
|
|
|
|
|
radix / BLIST_RADIX);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (scan[i].bm_bitmap == 0)
|
|
|
|
|
scan->bm_bitmap &= ~((u_daddr_t)1 << digit);
|
|
|
|
|
allocBlk = blk;
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
2017-07-24 17:23:53 +00:00
|
|
|
return (nblks);
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
#ifdef BLIST_DEBUG
|
|
|
|
|
|
2017-07-17 23:16:33 +00:00
|
|
|
static void
|
2017-08-01 03:51:26 +00:00
|
|
|
blst_radix_print(blmeta_t *scan, daddr_t blk, daddr_t radix, int tab)
|
1999-01-21 08:11:06 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t skip;
|
2019-05-11 09:09:10 +00:00
|
|
|
u_daddr_t mask;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
int digit;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2020-07-25 18:29:10 +00:00
|
|
|
if (radix == 1) {
|
1999-01-21 08:11:06 +00:00
|
|
|
printf(
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
"%*.*s(%08llx,%lld): bitmap %0*llx big=%lld\n",
|
1999-01-21 08:11:06 +00:00
|
|
|
tab, tab, "",
|
2020-07-25 18:29:10 +00:00
|
|
|
(long long)blk, (long long)BLIST_RADIX,
|
|
|
|
|
(int)(1 + (BLIST_RADIX - 1) / 4),
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
(long long)scan->bm_bitmap,
|
2002-12-15 19:17:57 +00:00
|
|
|
(long long)scan->bm_bighint
|
1999-01-21 08:11:06 +00:00
|
|
|
);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
printf(
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
"%*.*s(%08llx): subtree (%lld/%lld) bitmap %0*llx big=%lld {\n",
|
1999-01-21 08:11:06 +00:00
|
|
|
tab, tab, "",
|
2020-07-25 18:29:10 +00:00
|
|
|
(long long)blk, (long long)radix * BLIST_RADIX,
|
|
|
|
|
(long long)radix * BLIST_RADIX,
|
|
|
|
|
(int)(1 + (BLIST_RADIX - 1) / 4),
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
(long long)scan->bm_bitmap,
|
2002-12-15 19:17:57 +00:00
|
|
|
(long long)scan->bm_bighint
|
1999-01-21 08:11:06 +00:00
|
|
|
);
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
skip = radix_to_skip(radix);
|
1999-01-21 08:11:06 +00:00
|
|
|
tab += 4;
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = scan->bm_bitmap;
|
|
|
|
|
/* Examine the nonempty subtree associated with each bit set in mask */
|
|
|
|
|
do {
|
2019-05-11 09:09:10 +00:00
|
|
|
digit = bitpos(mask);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
blst_radix_print(&scan[1 + digit * skip], blk + digit * radix,
|
2020-07-25 18:29:10 +00:00
|
|
|
radix / BLIST_RADIX, tab);
|
2019-05-11 09:09:10 +00:00
|
|
|
} while ((mask ^= bitrange(digit, 1)) != 0);
|
1999-01-21 08:11:06 +00:00
|
|
|
tab -= 4;
|
|
|
|
|
|
|
|
|
|
printf(
|
|
|
|
|
"%*.*s}\n",
|
|
|
|
|
tab, tab, ""
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef BLIST_DEBUG
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
main(int ac, char **av)
|
|
|
|
|
{
|
2020-07-25 18:29:10 +00:00
|
|
|
daddr_t size = BLIST_RADIX * BLIST_RADIX;
|
1999-01-21 08:11:06 +00:00
|
|
|
int i;
|
|
|
|
|
blist_t bl;
|
2017-09-10 17:46:03 +00:00
|
|
|
struct sbuf *s;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
for (i = 1; i < ac; ++i) {
|
|
|
|
|
const char *ptr = av[i];
|
|
|
|
|
if (*ptr != '-') {
|
2020-07-25 18:29:10 +00:00
|
|
|
size = strtoll(ptr, NULL, 0);
|
1999-01-21 08:11:06 +00:00
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
ptr += 2;
|
|
|
|
|
fprintf(stderr, "Bad option: %s\n", ptr - 2);
|
|
|
|
|
exit(1);
|
|
|
|
|
}
|
2008-05-05 19:48:54 +00:00
|
|
|
bl = blist_create(size, M_WAITOK);
|
2020-07-25 18:29:10 +00:00
|
|
|
if (bl == NULL) {
|
|
|
|
|
fprintf(stderr, "blist_create failed\n");
|
|
|
|
|
exit(1);
|
|
|
|
|
}
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(bl, 0, size);
|
|
|
|
|
|
|
|
|
|
for (;;) {
|
|
|
|
|
char buf[1024];
|
2017-06-07 16:04:34 +00:00
|
|
|
long long da = 0;
|
2019-05-11 16:15:13 +00:00
|
|
|
int count = 0, maxcount = 0;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
printf("%lld/%lld/%lld> ", (long long)blist_avail(bl),
|
2020-07-25 18:29:10 +00:00
|
|
|
(long long)size, (long long)bl->bl_radix * BLIST_RADIX);
|
1999-01-21 08:11:06 +00:00
|
|
|
fflush(stdout);
|
|
|
|
|
if (fgets(buf, sizeof(buf), stdin) == NULL)
|
|
|
|
|
break;
|
|
|
|
|
switch(buf[0]) {
|
|
|
|
|
case 'r':
|
2019-05-11 16:15:13 +00:00
|
|
|
if (sscanf(buf + 1, "%d", &count) == 1) {
|
2017-06-07 16:04:34 +00:00
|
|
|
blist_resize(&bl, count, 1, M_WAITOK);
|
1999-01-21 08:11:06 +00:00
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
case 'p':
|
|
|
|
|
blist_print(bl);
|
|
|
|
|
break;
|
2017-09-10 17:46:03 +00:00
|
|
|
case 's':
|
|
|
|
|
s = sbuf_new_auto();
|
|
|
|
|
blist_stats(bl, s);
|
|
|
|
|
sbuf_finish(s);
|
|
|
|
|
printf("%s", sbuf_data(s));
|
|
|
|
|
sbuf_delete(s);
|
|
|
|
|
break;
|
1999-01-21 08:11:06 +00:00
|
|
|
case 'a':
|
2019-05-11 16:15:13 +00:00
|
|
|
if (sscanf(buf + 1, "%d%d", &count, &maxcount) == 2) {
|
|
|
|
|
daddr_t blk = blist_alloc(bl, &count, maxcount);
|
|
|
|
|
printf(" R=%08llx, c=%08d\n",
|
|
|
|
|
(long long)blk, count);
|
1999-01-21 08:11:06 +00:00
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
case 'f':
|
2019-05-11 16:15:13 +00:00
|
|
|
if (sscanf(buf + 1, "%llx %d", &da, &count) == 2) {
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(bl, da, count);
|
|
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
break;
|
2002-12-15 19:17:57 +00:00
|
|
|
case 'l':
|
2019-05-11 16:15:13 +00:00
|
|
|
if (sscanf(buf + 1, "%llx %d", &da, &count) == 2) {
|
2017-06-09 16:19:24 +00:00
|
|
|
printf(" n=%jd\n",
|
|
|
|
|
(intmax_t)blist_fill(bl, da, count));
|
2002-12-15 19:17:57 +00:00
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
break;
|
1999-01-21 08:11:06 +00:00
|
|
|
case '?':
|
|
|
|
|
case 'h':
|
|
|
|
|
puts(
|
|
|
|
|
"p -print\n"
|
2017-09-10 17:46:03 +00:00
|
|
|
"s -stats\n"
|
2019-05-11 16:15:13 +00:00
|
|
|
"a %d %d -allocate\n"
|
1999-01-21 08:11:06 +00:00
|
|
|
"f %x %d -free\n"
|
2002-12-15 19:17:57 +00:00
|
|
|
"l %x %d -fill\n"
|
1999-01-21 08:11:06 +00:00
|
|
|
"r %d -resize\n"
|
2019-05-10 22:02:29 +00:00
|
|
|
"h/? -help\n"
|
|
|
|
|
"q -quit"
|
1999-01-21 08:11:06 +00:00
|
|
|
);
|
|
|
|
|
break;
|
2019-05-10 22:02:29 +00:00
|
|
|
case 'q':
|
|
|
|
|
break;
|
1999-01-21 08:11:06 +00:00
|
|
|
default:
|
|
|
|
|
printf("?\n");
|
|
|
|
|
break;
|
|
|
|
|
}
|
2019-05-10 22:02:29 +00:00
|
|
|
if (buf[0] == 'q')
|
|
|
|
|
break;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
2019-05-10 18:22:40 +00:00
|
|
|
return (0);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif
|
