2003-08-12 23:24:05 +00:00
|
|
|
/*-
|
2017-11-20 19:43:44 +00:00
|
|
|
* SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
|
*
|
2003-08-12 23:24:05 +00:00
|
|
|
* Copyright (c) 1998 Matthew Dillon. All Rights Reserved.
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2016-09-15 13:16:20 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
2003-08-12 23:24:05 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
|
* without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS
|
|
|
|
|
* OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
|
|
|
|
|
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
|
|
|
|
|
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
|
|
|
|
* GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
|
|
|
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
|
|
|
|
|
* WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
|
|
|
|
|
* NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
|
|
|
|
|
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
|
*/
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* BLIST.C - Bitmap allocator/deallocator, using a radix tree with hinting
|
|
|
|
|
*
|
|
|
|
|
* This module implements a general bitmap allocator/deallocator. The
|
2017-07-17 23:16:33 +00:00
|
|
|
* allocator eats around 2 bits per 'block'. The module does not
|
|
|
|
|
* try to interpret the meaning of a 'block' other than to return
|
1999-01-21 08:11:06 +00:00
|
|
|
* SWAPBLK_NONE on an allocation failure.
|
|
|
|
|
*
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* A radix tree controls access to pieces of the bitmap, and includes
|
|
|
|
|
* auxiliary information at each interior node about the availabilty of
|
|
|
|
|
* contiguous free blocks in the subtree rooted at that node. Two radix
|
|
|
|
|
* constants are involved: one for the size of the bitmaps contained in the
|
|
|
|
|
* leaf nodes (BLIST_BMAP_RADIX), and one for the number of descendents of
|
|
|
|
|
* each of the meta (interior) nodes (BLIST_META_RADIX). Each subtree is
|
|
|
|
|
* associated with a range of blocks. The root of any subtree stores a
|
|
|
|
|
* hint field that defines an upper bound on the size of the largest
|
|
|
|
|
* allocation that can begin in the associated block range. A hint is an
|
|
|
|
|
* upper bound on a potential allocation, but not necessarily a tight upper
|
|
|
|
|
* bound.
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* The bitmap field in each node directs the search for available blocks.
|
|
|
|
|
* For a leaf node, a bit is set if the corresponding block is free. For a
|
|
|
|
|
* meta node, a bit is set if the corresponding subtree contains a free
|
|
|
|
|
* block somewhere within it. The search at a meta node considers only
|
|
|
|
|
* children of that node that represent a range that includes a free block.
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
|
|
|
|
* The hinting greatly increases code efficiency for allocations while
|
|
|
|
|
* the general radix structure optimizes both allocations and frees. The
|
2017-07-17 23:16:33 +00:00
|
|
|
* radix tree should be able to operate well no matter how much
|
1999-01-21 08:11:06 +00:00
|
|
|
* fragmentation there is and no matter how large a bitmap is used.
|
|
|
|
|
*
|
2013-02-05 20:08:33 +00:00
|
|
|
* The blist code wires all necessary memory at creation time. Neither
|
|
|
|
|
* allocations nor frees require interaction with the memory subsystem.
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* The non-blocking nature of allocations and frees is required by swap
|
|
|
|
|
* code (vm/swap_pager.c).
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* LAYOUT: The radix tree is laid out recursively using a linear array.
|
|
|
|
|
* Each meta node is immediately followed (laid out sequentially in
|
|
|
|
|
* memory) by BLIST_META_RADIX lower level nodes. This is a recursive
|
|
|
|
|
* structure but one that can be easily scanned through a very simple
|
|
|
|
|
* 'skip' calculation. The memory allocation is only large enough to
|
|
|
|
|
* cover the number of blocks requested at creation time. Nodes that
|
|
|
|
|
* represent blocks beyond that limit, nodes that would never be read
|
|
|
|
|
* or written, are not allocated, so that the last of the
|
|
|
|
|
* BLIST_META_RADIX lower level nodes of a some nodes may not be
|
|
|
|
|
* allocated.
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
2017-07-17 23:16:33 +00:00
|
|
|
* NOTE: the allocator cannot currently allocate more than
|
|
|
|
|
* BLIST_BMAP_RADIX blocks per call. It will panic with 'allocation too
|
|
|
|
|
* large' if you try. This is an area that could use improvement. The
|
|
|
|
|
* radix is large enough that this restriction does not effect the swap
|
|
|
|
|
* system, though. Currently only the allocation code is affected by
|
1999-01-21 08:11:06 +00:00
|
|
|
* this algorithmic unfeature. The freeing code can handle arbitrary
|
|
|
|
|
* ranges.
|
|
|
|
|
*
|
|
|
|
|
* This code can be compiled stand-alone for debugging.
|
|
|
|
|
*/
|
|
|
|
|
|
2003-06-11 00:56:59 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifdef _KERNEL
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
#include <sys/param.h>
|
|
|
|
|
#include <sys/systm.h>
|
|
|
|
|
#include <sys/lock.h>
|
|
|
|
|
#include <sys/kernel.h>
|
|
|
|
|
#include <sys/blist.h>
|
|
|
|
|
#include <sys/malloc.h>
|
2017-09-10 17:46:03 +00:00
|
|
|
#include <sys/sbuf.h>
|
2001-07-04 16:20:28 +00:00
|
|
|
#include <sys/proc.h>
|
2017-07-17 23:16:33 +00:00
|
|
|
#include <sys/mutex.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
#else
|
|
|
|
|
|
|
|
|
|
#ifndef BLIST_NO_DEBUG
|
|
|
|
|
#define BLIST_DEBUG
|
|
|
|
|
#endif
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
#include <sys/errno.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
#include <sys/types.h>
|
2017-06-07 16:04:34 +00:00
|
|
|
#include <sys/malloc.h>
|
2017-09-10 17:46:03 +00:00
|
|
|
#include <sys/sbuf.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
#include <stdio.h>
|
|
|
|
|
#include <string.h>
|
The blst_radix_init function has two purposes - to compute the number of
nodes to allocate for the blist, and to initialize them. The computation
can be done much more quickly by identifying the terminating node, if any,
at every level of the tree and then summing the number of nodes at each
level that precedes the topmost terminator. The initialization can also be
done quickly, since settings at the root mark the tree as all-allocated, and
only a few terminator nodes need to be marked in the rest of the tree.
Eliminate blst_radix_init, and perform its two functions more simply in
blist_create.
The allocation of the blist takes places in two pieces, but there's no good
reason to do so, when a single allocation is sufficient, and simpler.
Allocate the blist struct, and the array of nodes associated with it, with a
single allocation.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11968
2017-10-08 22:17:39 +00:00
|
|
|
#include <stddef.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
#include <stdlib.h>
|
|
|
|
|
#include <stdarg.h>
|
2017-06-27 17:45:26 +00:00
|
|
|
#include <stdbool.h>
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
#define bitcount64(x) __bitcount64((uint64_t)(x))
|
2002-12-15 19:17:57 +00:00
|
|
|
#define malloc(a,b,c) calloc(a, 1)
|
1999-01-21 08:11:06 +00:00
|
|
|
#define free(a,b) free(a)
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
#define ummin(a,b) ((a) < (b) ? (a) : (b))
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
#include <sys/blist.h>
|
|
|
|
|
|
|
|
|
|
void panic(const char *ctl, ...);
|
|
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* static support functions
|
|
|
|
|
*/
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
static daddr_t blst_leaf_alloc(blmeta_t *scan, daddr_t blk, int count);
|
2017-08-13 16:39:49 +00:00
|
|
|
static daddr_t blst_meta_alloc(blmeta_t *scan, daddr_t cursor, daddr_t count,
|
|
|
|
|
u_daddr_t radix);
|
1999-01-21 08:11:06 +00:00
|
|
|
static void blst_leaf_free(blmeta_t *scan, daddr_t relblk, int count);
|
2017-07-17 23:16:33 +00:00
|
|
|
static void blst_meta_free(blmeta_t *scan, daddr_t freeBlk, daddr_t count,
|
2017-08-13 16:39:49 +00:00
|
|
|
u_daddr_t radix);
|
2017-07-17 23:16:33 +00:00
|
|
|
static void blst_copy(blmeta_t *scan, daddr_t blk, daddr_t radix,
|
2017-08-01 03:51:26 +00:00
|
|
|
blist_t dest, daddr_t count);
|
2017-06-09 16:19:24 +00:00
|
|
|
static daddr_t blst_leaf_fill(blmeta_t *scan, daddr_t blk, int count);
|
|
|
|
|
static daddr_t blst_meta_fill(blmeta_t *scan, daddr_t allocBlk, daddr_t count,
|
2017-08-13 16:39:49 +00:00
|
|
|
u_daddr_t radix);
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifndef _KERNEL
|
2017-06-27 17:45:26 +00:00
|
|
|
static void blst_radix_print(blmeta_t *scan, daddr_t blk, daddr_t radix,
|
2017-08-01 03:51:26 +00:00
|
|
|
int tab);
|
1999-01-21 08:11:06 +00:00
|
|
|
#endif
|
|
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifdef _KERNEL
|
1999-01-21 08:11:06 +00:00
|
|
|
static MALLOC_DEFINE(M_SWAP, "SWAP", "Swap space");
|
|
|
|
|
#endif
|
|
|
|
|
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
_Static_assert(BLIST_BMAP_RADIX % BLIST_META_RADIX == 0,
|
|
|
|
|
"radix divisibility error");
|
|
|
|
|
#define BLIST_BMAP_MASK (BLIST_BMAP_RADIX - 1)
|
2017-09-10 17:46:03 +00:00
|
|
|
#define BLIST_META_MASK (BLIST_META_RADIX - 1)
|
2017-08-04 04:23:23 +00:00
|
|
|
|
2017-08-01 03:51:26 +00:00
|
|
|
/*
|
|
|
|
|
* For a subtree that can represent the state of up to 'radix' blocks, the
|
|
|
|
|
* number of leaf nodes of the subtree is L=radix/BLIST_BMAP_RADIX. If 'm'
|
|
|
|
|
* is short for BLIST_META_RADIX, then for a tree of height h with L=m**h
|
|
|
|
|
* leaf nodes, the total number of tree nodes is 1 + m + m**2 + ... + m**h,
|
|
|
|
|
* or, equivalently, (m**(h+1)-1)/(m-1). This quantity is called 'skip'
|
|
|
|
|
* in the 'meta' functions that process subtrees. Since integer division
|
|
|
|
|
* discards remainders, we can express this computation as
|
|
|
|
|
* skip = (m * m**h) / (m - 1)
|
2017-08-04 04:23:23 +00:00
|
|
|
* skip = (m * (radix / BLIST_BMAP_RADIX)) / (m - 1)
|
|
|
|
|
* and since m divides BLIST_BMAP_RADIX, we can simplify further to
|
|
|
|
|
* skip = (radix / (BLIST_BMAP_RADIX / m)) / (m - 1)
|
|
|
|
|
* skip = radix / ((BLIST_BMAP_RADIX / m) * (m - 1))
|
|
|
|
|
* so that simple integer division by a constant can safely be used for the
|
|
|
|
|
* calculation.
|
2017-08-01 03:51:26 +00:00
|
|
|
*/
|
|
|
|
|
static inline daddr_t
|
|
|
|
|
radix_to_skip(daddr_t radix)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (radix /
|
2017-09-10 17:46:03 +00:00
|
|
|
((BLIST_BMAP_RADIX / BLIST_META_RADIX) * BLIST_META_MASK));
|
|
|
|
|
}
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
/*
|
|
|
|
|
* Provide a mask with count bits set, starting as position n.
|
|
|
|
|
*/
|
|
|
|
|
static inline u_daddr_t
|
|
|
|
|
bitrange(int n, int count)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (((u_daddr_t)-1 << n) &
|
|
|
|
|
((u_daddr_t)-1 >> (BLIST_BMAP_RADIX - (n + count))));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2017-09-10 17:46:03 +00:00
|
|
|
/*
|
|
|
|
|
* Use binary search, or a faster method, to find the 1 bit in a u_daddr_t.
|
|
|
|
|
* Assumes that the argument has only one bit set.
|
|
|
|
|
*/
|
|
|
|
|
static inline int
|
|
|
|
|
bitpos(u_daddr_t mask)
|
|
|
|
|
{
|
|
|
|
|
int hi, lo, mid;
|
|
|
|
|
|
|
|
|
|
switch (sizeof(mask)) {
|
|
|
|
|
#ifdef HAVE_INLINE_FFSLL
|
|
|
|
|
case sizeof(long long):
|
|
|
|
|
return (ffsll(mask) - 1);
|
|
|
|
|
#endif
|
|
|
|
|
default:
|
|
|
|
|
lo = 0;
|
|
|
|
|
hi = BLIST_BMAP_RADIX;
|
|
|
|
|
while (lo + 1 < hi) {
|
|
|
|
|
mid = (lo + hi) >> 1;
|
|
|
|
|
if ((mask >> mid) != 0)
|
|
|
|
|
lo = mid;
|
|
|
|
|
else
|
|
|
|
|
hi = mid;
|
|
|
|
|
}
|
|
|
|
|
return (lo);
|
|
|
|
|
}
|
2017-08-01 03:51:26 +00:00
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* blist_create() - create a blist capable of handling up to the specified
|
|
|
|
|
* number of blocks
|
|
|
|
|
*
|
2011-12-03 15:41:37 +00:00
|
|
|
* blocks - must be greater than 0
|
2008-05-05 19:48:54 +00:00
|
|
|
* flags - malloc flags
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
2017-07-17 23:16:33 +00:00
|
|
|
* The smallest blist consists of a single leaf node capable of
|
1999-01-21 08:11:06 +00:00
|
|
|
* managing BLIST_BMAP_RADIX blocks.
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
blist_t
|
2008-05-05 19:48:54 +00:00
|
|
|
blist_create(daddr_t blocks, int flags)
|
1999-01-21 08:11:06 +00:00
|
|
|
{
|
|
|
|
|
blist_t bl;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
u_daddr_t nodes, radix;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2018-09-05 19:05:30 +00:00
|
|
|
if (blocks == 0)
|
|
|
|
|
panic("invalid block count");
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
2018-09-05 19:05:30 +00:00
|
|
|
* Calculate the radix and node count used for scanning.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
nodes = 1;
|
1999-01-21 08:11:06 +00:00
|
|
|
radix = BLIST_BMAP_RADIX;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
while (radix <= blocks) {
|
|
|
|
|
nodes += 1 + (blocks - 1) / radix;
|
2003-01-11 01:09:51 +00:00
|
|
|
radix *= BLIST_META_RADIX;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
2017-10-09 18:19:06 +00:00
|
|
|
bl = malloc(offsetof(struct blist, bl_root[nodes]), M_SWAP, flags |
|
|
|
|
|
M_ZERO);
|
2017-06-10 16:11:39 +00:00
|
|
|
if (bl == NULL)
|
|
|
|
|
return (NULL);
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
bl->bl_blocks = blocks;
|
|
|
|
|
bl->bl_radix = radix;
|
|
|
|
|
|
|
|
|
|
#if defined(BLIST_DEBUG)
|
|
|
|
|
printf(
|
2002-12-15 19:17:57 +00:00
|
|
|
"BLIST representing %lld blocks (%lld MB of swap)"
|
|
|
|
|
", requiring %lldK of ram\n",
|
|
|
|
|
(long long)bl->bl_blocks,
|
|
|
|
|
(long long)bl->bl_blocks * 4 / 1024,
|
2017-06-10 16:11:39 +00:00
|
|
|
(long long)(nodes * sizeof(blmeta_t) + 1023) / 1024
|
1999-01-21 08:11:06 +00:00
|
|
|
);
|
2002-12-15 19:17:57 +00:00
|
|
|
printf("BLIST raw radix tree contains %lld records\n",
|
2017-06-10 16:11:39 +00:00
|
|
|
(long long)nodes);
|
1999-01-21 08:11:06 +00:00
|
|
|
#endif
|
|
|
|
|
|
2017-06-10 16:11:39 +00:00
|
|
|
return (bl);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
2017-07-17 23:16:33 +00:00
|
|
|
void
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_destroy(blist_t bl)
|
|
|
|
|
{
|
The blst_radix_init function has two purposes - to compute the number of
nodes to allocate for the blist, and to initialize them. The computation
can be done much more quickly by identifying the terminating node, if any,
at every level of the tree and then summing the number of nodes at each
level that precedes the topmost terminator. The initialization can also be
done quickly, since settings at the root mark the tree as all-allocated, and
only a few terminator nodes need to be marked in the rest of the tree.
Eliminate blst_radix_init, and perform its two functions more simply in
blist_create.
The allocation of the blist takes places in two pieces, but there's no good
reason to do so, when a single allocation is sufficient, and simpler.
Allocate the blist struct, and the array of nodes associated with it, with a
single allocation.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11968
2017-10-08 22:17:39 +00:00
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
free(bl, M_SWAP);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
2017-06-13 17:49:49 +00:00
|
|
|
* blist_alloc() - reserve space in the block bitmap. Return the base
|
1999-01-21 08:11:06 +00:00
|
|
|
* of a contiguous region or SWAPBLK_NONE if space could
|
|
|
|
|
* not be allocated.
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
daddr_t
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_alloc(blist_t bl, daddr_t count)
|
|
|
|
|
{
|
2017-06-13 17:49:49 +00:00
|
|
|
daddr_t blk;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (count > BLIST_MAX_ALLOC)
|
|
|
|
|
panic("allocation too large");
|
|
|
|
|
|
2017-06-18 18:23:39 +00:00
|
|
|
/*
|
|
|
|
|
* This loop iterates at most twice. An allocation failure in the
|
|
|
|
|
* first iteration leads to a second iteration only if the cursor was
|
|
|
|
|
* non-zero. When the cursor is zero, an allocation failure will
|
|
|
|
|
* reduce the hint, stopping further iterations.
|
|
|
|
|
*/
|
|
|
|
|
while (count <= bl->bl_root->bm_bighint) {
|
2017-08-13 16:39:49 +00:00
|
|
|
blk = blst_meta_alloc(bl->bl_root, bl->bl_cursor, count,
|
|
|
|
|
bl->bl_radix);
|
2017-06-18 18:23:39 +00:00
|
|
|
if (blk != SWAPBLK_NONE) {
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
bl->bl_avail -= count;
|
2017-06-18 18:23:39 +00:00
|
|
|
bl->bl_cursor = blk + count;
|
2017-08-25 18:47:23 +00:00
|
|
|
if (bl->bl_cursor == bl->bl_blocks)
|
|
|
|
|
bl->bl_cursor = 0;
|
2017-06-18 18:23:39 +00:00
|
|
|
return (blk);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
}
|
|
|
|
|
bl->bl_cursor = 0;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
2017-06-13 17:49:49 +00:00
|
|
|
return (SWAPBLK_NONE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_avail() - return the number of free blocks.
|
|
|
|
|
*/
|
|
|
|
|
daddr_t
|
|
|
|
|
blist_avail(blist_t bl)
|
|
|
|
|
{
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
return (bl->bl_avail);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_free() - free up space in the block bitmap. Return the base
|
|
|
|
|
* of a contiguous region. Panic if an inconsistancy is
|
|
|
|
|
* found.
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
void
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(blist_t bl, daddr_t blkno, daddr_t count)
|
|
|
|
|
{
|
2017-07-24 17:23:53 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (blkno < 0 || blkno + count > bl->bl_blocks)
|
|
|
|
|
panic("freeing invalid range");
|
2017-08-13 16:39:49 +00:00
|
|
|
blst_meta_free(bl->bl_root, blkno, count, bl->bl_radix);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
bl->bl_avail += count;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
2002-12-15 19:17:57 +00:00
|
|
|
/*
|
|
|
|
|
* blist_fill() - mark a region in the block bitmap as off-limits
|
|
|
|
|
* to the allocator (i.e. allocate it), ignoring any
|
|
|
|
|
* existing allocations. Return the number of blocks
|
|
|
|
|
* actually filled that were free before the call.
|
|
|
|
|
*/
|
2017-06-09 16:19:24 +00:00
|
|
|
daddr_t
|
2002-12-15 19:17:57 +00:00
|
|
|
blist_fill(blist_t bl, daddr_t blkno, daddr_t count)
|
|
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t filled;
|
2002-12-15 19:17:57 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (blkno < 0 || blkno + count > bl->bl_blocks)
|
|
|
|
|
panic("filling invalid range");
|
|
|
|
|
filled = blst_meta_fill(bl->bl_root, blkno, count, bl->bl_radix);
|
|
|
|
|
bl->bl_avail -= filled;
|
|
|
|
|
return (filled);
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* blist_resize() - resize an existing radix tree to handle the
|
|
|
|
|
* specified number of blocks. This will reallocate
|
|
|
|
|
* the tree and transfer the previous bitmap to the new
|
|
|
|
|
* one. When extending the tree you can specify whether
|
|
|
|
|
* the new blocks are to left allocated or freed.
|
|
|
|
|
*/
|
|
|
|
|
void
|
2008-05-05 19:48:54 +00:00
|
|
|
blist_resize(blist_t *pbl, daddr_t count, int freenew, int flags)
|
1999-01-21 08:11:06 +00:00
|
|
|
{
|
2008-05-05 19:48:54 +00:00
|
|
|
blist_t newbl = blist_create(count, flags);
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_t save = *pbl;
|
|
|
|
|
|
|
|
|
|
*pbl = newbl;
|
|
|
|
|
if (count > save->bl_blocks)
|
|
|
|
|
count = save->bl_blocks;
|
2017-08-01 03:51:26 +00:00
|
|
|
blst_copy(save->bl_root, 0, save->bl_radix, newbl, count);
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If resizing upwards, should we free the new space or not?
|
|
|
|
|
*/
|
|
|
|
|
if (freenew && count < newbl->bl_blocks) {
|
|
|
|
|
blist_free(newbl, count, newbl->bl_blocks - count);
|
|
|
|
|
}
|
|
|
|
|
blist_destroy(save);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#ifdef BLIST_DEBUG
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_print() - dump radix tree
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
blist_print(blist_t bl)
|
|
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
printf("BLIST avail = %jd, cursor = %08jx {\n",
|
|
|
|
|
(uintmax_t)bl->bl_avail, (uintmax_t)bl->bl_cursor);
|
|
|
|
|
|
|
|
|
|
if (bl->bl_root->bm_bitmap != 0)
|
|
|
|
|
blst_radix_print(bl->bl_root, 0, bl->bl_radix, 4);
|
1999-01-21 08:11:06 +00:00
|
|
|
printf("}\n");
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
2017-09-10 17:46:03 +00:00
|
|
|
static const u_daddr_t fib[] = {
|
|
|
|
|
1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584,
|
|
|
|
|
4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811,
|
|
|
|
|
514229, 832040, 1346269, 2178309, 3524578,
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Use 'gap' to describe a maximal range of unallocated blocks/bits.
|
|
|
|
|
*/
|
|
|
|
|
struct gap_stats {
|
|
|
|
|
daddr_t start; /* current gap start, or SWAPBLK_NONE */
|
|
|
|
|
daddr_t num; /* number of gaps observed */
|
|
|
|
|
daddr_t max; /* largest gap size */
|
|
|
|
|
daddr_t avg; /* average gap size */
|
|
|
|
|
daddr_t err; /* sum - num * avg */
|
|
|
|
|
daddr_t histo[nitems(fib)]; /* # gaps in each size range */
|
|
|
|
|
int max_bucket; /* last histo elt with nonzero val */
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* gap_stats_counting() - is the state 'counting 1 bits'?
|
|
|
|
|
* or 'skipping 0 bits'?
|
|
|
|
|
*/
|
|
|
|
|
static inline bool
|
|
|
|
|
gap_stats_counting(const struct gap_stats *stats)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
return (stats->start != SWAPBLK_NONE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* init_gap_stats() - initialize stats on gap sizes
|
|
|
|
|
*/
|
|
|
|
|
static inline void
|
|
|
|
|
init_gap_stats(struct gap_stats *stats)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
bzero(stats, sizeof(*stats));
|
|
|
|
|
stats->start = SWAPBLK_NONE;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* update_gap_stats() - update stats on gap sizes
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
update_gap_stats(struct gap_stats *stats, daddr_t posn)
|
|
|
|
|
{
|
|
|
|
|
daddr_t size;
|
|
|
|
|
int hi, lo, mid;
|
|
|
|
|
|
|
|
|
|
if (!gap_stats_counting(stats)) {
|
|
|
|
|
stats->start = posn;
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
size = posn - stats->start;
|
|
|
|
|
stats->start = SWAPBLK_NONE;
|
|
|
|
|
if (size > stats->max)
|
|
|
|
|
stats->max = size;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Find the fibonacci range that contains size,
|
|
|
|
|
* expecting to find it in an early range.
|
|
|
|
|
*/
|
|
|
|
|
lo = 0;
|
|
|
|
|
hi = 1;
|
|
|
|
|
while (hi < nitems(fib) && fib[hi] <= size) {
|
|
|
|
|
lo = hi;
|
|
|
|
|
hi *= 2;
|
|
|
|
|
}
|
|
|
|
|
if (hi >= nitems(fib))
|
|
|
|
|
hi = nitems(fib);
|
|
|
|
|
while (lo + 1 != hi) {
|
|
|
|
|
mid = (lo + hi) >> 1;
|
|
|
|
|
if (fib[mid] <= size)
|
|
|
|
|
lo = mid;
|
|
|
|
|
else
|
|
|
|
|
hi = mid;
|
|
|
|
|
}
|
|
|
|
|
stats->histo[lo]++;
|
|
|
|
|
if (lo > stats->max_bucket)
|
|
|
|
|
stats->max_bucket = lo;
|
|
|
|
|
stats->err += size - stats->avg;
|
|
|
|
|
stats->num++;
|
|
|
|
|
stats->avg += stats->err / stats->num;
|
|
|
|
|
stats->err %= stats->num;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* dump_gap_stats() - print stats on gap sizes
|
|
|
|
|
*/
|
|
|
|
|
static inline void
|
|
|
|
|
dump_gap_stats(const struct gap_stats *stats, struct sbuf *s)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
sbuf_printf(s, "number of maximal free ranges: %jd\n",
|
|
|
|
|
(intmax_t)stats->num);
|
|
|
|
|
sbuf_printf(s, "largest free range: %jd\n", (intmax_t)stats->max);
|
|
|
|
|
sbuf_printf(s, "average maximal free range size: %jd\n",
|
|
|
|
|
(intmax_t)stats->avg);
|
|
|
|
|
sbuf_printf(s, "number of maximal free ranges of different sizes:\n");
|
|
|
|
|
sbuf_printf(s, " count | size range\n");
|
|
|
|
|
sbuf_printf(s, " ----- | ----------\n");
|
|
|
|
|
for (i = 0; i < stats->max_bucket; i++) {
|
|
|
|
|
if (stats->histo[i] != 0) {
|
|
|
|
|
sbuf_printf(s, "%20jd | ",
|
|
|
|
|
(intmax_t)stats->histo[i]);
|
|
|
|
|
if (fib[i] != fib[i + 1] - 1)
|
|
|
|
|
sbuf_printf(s, "%jd to %jd\n", (intmax_t)fib[i],
|
|
|
|
|
(intmax_t)fib[i + 1] - 1);
|
|
|
|
|
else
|
|
|
|
|
sbuf_printf(s, "%jd\n", (intmax_t)fib[i]);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
sbuf_printf(s, "%20jd | ", (intmax_t)stats->histo[i]);
|
|
|
|
|
if (stats->histo[i] > 1)
|
|
|
|
|
sbuf_printf(s, "%jd to %jd\n", (intmax_t)fib[i],
|
|
|
|
|
(intmax_t)stats->max);
|
|
|
|
|
else
|
|
|
|
|
sbuf_printf(s, "%jd\n", (intmax_t)stats->max);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_stats() - dump radix tree stats
|
|
|
|
|
*/
|
|
|
|
|
void
|
|
|
|
|
blist_stats(blist_t bl, struct sbuf *s)
|
|
|
|
|
{
|
|
|
|
|
struct gap_stats gstats;
|
|
|
|
|
struct gap_stats *stats = &gstats;
|
|
|
|
|
daddr_t i, nodes, radix;
|
|
|
|
|
u_daddr_t bit, diff, mask;
|
|
|
|
|
|
|
|
|
|
init_gap_stats(stats);
|
|
|
|
|
nodes = 0;
|
|
|
|
|
i = bl->bl_radix;
|
|
|
|
|
while (i < bl->bl_radix + bl->bl_blocks) {
|
|
|
|
|
/*
|
|
|
|
|
* Find max size subtree starting at i.
|
|
|
|
|
*/
|
|
|
|
|
radix = BLIST_BMAP_RADIX;
|
|
|
|
|
while (((i / radix) & BLIST_META_MASK) == 0)
|
|
|
|
|
radix *= BLIST_META_RADIX;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Check for skippable subtrees starting at i.
|
|
|
|
|
*/
|
|
|
|
|
while (radix > BLIST_BMAP_RADIX) {
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (bl->bl_root[nodes].bm_bitmap == 0) {
|
2017-09-10 17:46:03 +00:00
|
|
|
if (gap_stats_counting(stats))
|
|
|
|
|
update_gap_stats(stats, i);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Skip subtree root.
|
|
|
|
|
*/
|
|
|
|
|
nodes++;
|
|
|
|
|
radix /= BLIST_META_RADIX;
|
|
|
|
|
}
|
|
|
|
|
if (radix == BLIST_BMAP_RADIX) {
|
|
|
|
|
/*
|
|
|
|
|
* Scan leaf.
|
|
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = bl->bl_root[nodes].bm_bitmap;
|
2017-09-10 17:46:03 +00:00
|
|
|
diff = mask ^ (mask << 1);
|
|
|
|
|
if (gap_stats_counting(stats))
|
|
|
|
|
diff ^= 1;
|
|
|
|
|
while (diff != 0) {
|
|
|
|
|
bit = diff & -diff;
|
|
|
|
|
update_gap_stats(stats, i + bitpos(bit));
|
|
|
|
|
diff ^= bit;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
nodes += radix_to_skip(radix);
|
|
|
|
|
i += radix;
|
|
|
|
|
}
|
|
|
|
|
update_gap_stats(stats, i);
|
|
|
|
|
dump_gap_stats(stats, s);
|
|
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/************************************************************************
|
|
|
|
|
* ALLOCATION SUPPORT FUNCTIONS *
|
|
|
|
|
************************************************************************
|
|
|
|
|
*
|
2017-07-17 23:16:33 +00:00
|
|
|
* These support functions do all the actual work. They may seem
|
1999-01-21 08:11:06 +00:00
|
|
|
* rather longish, but that's because I've commented them up. The
|
|
|
|
|
* actual code is straight forward.
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* BLST_NEXT_LEAF_ALLOC() - allocate the first few blocks in the next leaf.
|
|
|
|
|
*
|
|
|
|
|
* 'scan' is a leaf node, associated with a block containing 'blk'.
|
|
|
|
|
* The next leaf node could be adjacent, or several nodes away if the
|
|
|
|
|
* least common ancestor of 'scan' and its neighbor is several levels
|
|
|
|
|
* up. Use 'blk' to determine how many meta-nodes lie between the
|
|
|
|
|
* leaves. If the next leaf has enough initial bits set, clear them
|
|
|
|
|
* and clear the bits in the meta nodes on the path up to the least
|
|
|
|
|
* common ancestor to mark any subtrees made completely empty.
|
|
|
|
|
*/
|
|
|
|
|
static int
|
|
|
|
|
blst_next_leaf_alloc(blmeta_t *scan, daddr_t blk, int count)
|
|
|
|
|
{
|
|
|
|
|
blmeta_t *next;
|
|
|
|
|
daddr_t skip;
|
|
|
|
|
u_daddr_t radix;
|
|
|
|
|
int digit;
|
|
|
|
|
|
|
|
|
|
next = scan + 1;
|
|
|
|
|
blk += BLIST_BMAP_RADIX;
|
|
|
|
|
radix = BLIST_BMAP_RADIX;
|
|
|
|
|
while ((digit = ((blk / radix) & BLIST_META_MASK)) == 0 &&
|
|
|
|
|
(next->bm_bitmap & 1) == 1) {
|
|
|
|
|
next++;
|
|
|
|
|
radix *= BLIST_META_RADIX;
|
|
|
|
|
}
|
|
|
|
|
if (((next->bm_bitmap + 1) & ~((u_daddr_t)-1 << count)) != 0) {
|
|
|
|
|
/*
|
|
|
|
|
* The next leaf doesn't have enough free blocks at the
|
|
|
|
|
* beginning to complete the spanning allocation.
|
|
|
|
|
*/
|
|
|
|
|
return (ENOMEM);
|
|
|
|
|
}
|
|
|
|
|
/* Clear the first 'count' bits in the next leaf to allocate. */
|
|
|
|
|
next->bm_bitmap &= (u_daddr_t)-1 << count;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Update bitmaps of next-ancestors, up to least common ancestor.
|
|
|
|
|
*/
|
|
|
|
|
skip = radix_to_skip(radix);
|
|
|
|
|
while (radix != BLIST_BMAP_RADIX && next->bm_bitmap == 0) {
|
|
|
|
|
(--next)->bm_bitmap ^= 1;
|
|
|
|
|
radix /= BLIST_META_RADIX;
|
|
|
|
|
}
|
|
|
|
|
if (next->bm_bitmap == 0)
|
|
|
|
|
scan[-digit * skip].bm_bitmap ^= (u_daddr_t)1 << digit;
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLST_LEAF_ALLOC() - allocate at a leaf in the radix tree (a bitmap).
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
* This is the core of the allocator and is optimized for the
|
|
|
|
|
* BLIST_BMAP_RADIX block allocation case. Otherwise, execution
|
2017-09-10 17:46:03 +00:00
|
|
|
* time is proportional to log2(count) + bitpos time.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
|
|
|
|
static daddr_t
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
blst_leaf_alloc(blmeta_t *scan, daddr_t blk, int count)
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
{
|
|
|
|
|
u_daddr_t mask;
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
int count1, hi, lo, num_shifts, range1, range_ext;
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
|
|
|
|
|
range1 = 0;
|
|
|
|
|
count1 = count - 1;
|
|
|
|
|
num_shifts = fls(count1);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = scan->bm_bitmap;
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
while ((-mask & ~mask) != 0 && num_shifts > 0) {
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
* If bit i is set in mask, then bits in [i, i+range1] are set
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* in scan->bm_bitmap. The value of range1 is equal to
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
* count1 >> num_shifts. Grow range and reduce num_shifts to 0,
|
|
|
|
|
* while preserving these invariants. The updates to mask leave
|
|
|
|
|
* fewer bits set, but each bit that remains set represents a
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* longer string of consecutive bits set in scan->bm_bitmap.
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* If more updates to mask cannot clear more bits, because mask
|
|
|
|
|
* is partitioned with all 0 bits preceding all 1 bits, the loop
|
|
|
|
|
* terminates immediately.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
num_shifts--;
|
|
|
|
|
range_ext = range1 + ((count1 >> num_shifts) & 1);
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
/*
|
|
|
|
|
* mask is a signed quantity for the shift because when it is
|
|
|
|
|
* shifted right, the sign bit should copied; when the last
|
|
|
|
|
* block of the leaf is free, pretend, for a while, that all the
|
|
|
|
|
* blocks that follow it are also free.
|
|
|
|
|
*/
|
|
|
|
|
mask &= (daddr_t)mask >> range_ext;
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
range1 += range_ext;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
if (mask == 0) {
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
* Update bighint. There is no allocation bigger than range1
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* starting in this leaf.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
scan->bm_bighint = range1;
|
|
|
|
|
return (SWAPBLK_NONE);
|
|
|
|
|
}
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
/* Discard any candidates that appear before blk. */
|
|
|
|
|
mask &= (u_daddr_t)-1 << (blk & BLIST_BMAP_MASK);
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
if (mask == 0)
|
|
|
|
|
return (SWAPBLK_NONE);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The least significant set bit in mask marks the start of the first
|
|
|
|
|
* available range of sufficient size. Clear all the bits but that one,
|
2017-09-10 17:46:03 +00:00
|
|
|
* and then find its position.
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
*/
|
|
|
|
|
mask &= -mask;
|
2017-09-10 17:46:03 +00:00
|
|
|
lo = bitpos(mask);
|
Change blst_leaf_alloc() to handle a cursor argument, and to improve
performance.
To find in the leaf bitmap all ranges of sufficient length, use a doubling
strategy with shift-and-and until each bit still set represents a bit
sequence of length 'count', or until the bitmask is zero. In the latter
case, update the hint based on the first bit sequence length not found to
be available. For example, seeking an interval of length 12, the set bits
of the bitmap would represent intervals of length 1, then 2, then 3, then
6, then 12. If no bits are set at the point when each bit represents an
interval of length 6, then the hint can be updated to 5 and the search
terminated.
If long-enough intervals are found, discard those before the cursor. If
any remain, use binary search to find the position of the first of them,
and allocate that interval.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: kib, markj
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D11426
2017-07-01 05:27:40 +00:00
|
|
|
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
hi = lo + count;
|
|
|
|
|
if (hi > BLIST_BMAP_RADIX) {
|
|
|
|
|
/*
|
|
|
|
|
* An allocation within this leaf is impossible, so a successful
|
|
|
|
|
* allocation depends on the next leaf providing some of the blocks.
|
|
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (blst_next_leaf_alloc(scan, blk, hi - BLIST_BMAP_RADIX) != 0)
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* The hint cannot be updated, because the same
|
|
|
|
|
* allocation request could be satisfied later, by this
|
|
|
|
|
* leaf, if the state of the next leaf changes, and
|
|
|
|
|
* without any changes to this leaf.
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
*/
|
|
|
|
|
return (SWAPBLK_NONE);
|
|
|
|
|
hi = BLIST_BMAP_RADIX;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Set the bits of mask at position 'lo' and higher. */
|
|
|
|
|
mask = -mask;
|
|
|
|
|
if (hi == BLIST_BMAP_RADIX) {
|
|
|
|
|
/*
|
|
|
|
|
* Update bighint. There is no allocation bigger than range1
|
|
|
|
|
* available in this leaf after this allocation completes.
|
|
|
|
|
*/
|
|
|
|
|
scan->bm_bighint = range1;
|
|
|
|
|
} else {
|
|
|
|
|
/* Clear the bits of mask at position 'hi' and higher. */
|
|
|
|
|
mask &= (u_daddr_t)-1 >> (BLIST_BMAP_RADIX - hi);
|
|
|
|
|
}
|
|
|
|
|
/* Clear the allocated bits from this leaf. */
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bitmap &= ~mask;
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
return ((blk & ~BLIST_BMAP_MASK) + lo);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* blist_meta_alloc() - allocate at a meta in the radix tree.
|
|
|
|
|
*
|
|
|
|
|
* Attempt to allocate at a meta node. If we can't, we update
|
|
|
|
|
* bighint and return a failure. Updating bighint optimize future
|
|
|
|
|
* calls that hit this node. We have to check for our collapse cases
|
|
|
|
|
* and we have a few optimizations strewn in as well.
|
|
|
|
|
*/
|
|
|
|
|
static daddr_t
|
2017-08-13 16:39:49 +00:00
|
|
|
blst_meta_alloc(blmeta_t *scan, daddr_t cursor, daddr_t count, u_daddr_t radix)
|
2017-06-18 18:23:39 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t blk, i, r, skip;
|
|
|
|
|
u_daddr_t bit, mask;
|
2017-06-18 18:23:39 +00:00
|
|
|
bool scan_from_start;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
int digit;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-07-24 17:23:53 +00:00
|
|
|
if (radix == BLIST_BMAP_RADIX)
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
return (blst_leaf_alloc(scan, cursor, count));
|
|
|
|
|
blk = cursor & -radix;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan_from_start = (cursor == blk);
|
|
|
|
|
radix /= BLIST_META_RADIX;
|
2017-08-01 03:51:26 +00:00
|
|
|
skip = radix_to_skip(radix);
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = scan->bm_bitmap;
|
|
|
|
|
|
|
|
|
|
/* Discard any candidates that appear before cursor. */
|
|
|
|
|
digit = (cursor / radix) & BLIST_META_MASK;
|
|
|
|
|
mask &= (u_daddr_t)-1 << digit;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* If the first try is for a block that includes the cursor, pre-undo
|
|
|
|
|
* the digit * radix offset in the first call; otherwise, ignore the
|
|
|
|
|
* cursor entirely.
|
2017-06-13 17:49:49 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (((mask >> digit) & 1) == 1)
|
|
|
|
|
cursor -= digit * radix;
|
|
|
|
|
else
|
|
|
|
|
cursor = blk;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
/*
|
|
|
|
|
* Examine the nonempty subtree associated with each bit set in mask.
|
|
|
|
|
*/
|
|
|
|
|
do {
|
|
|
|
|
bit = mask & -mask;
|
|
|
|
|
digit = bitpos(bit);
|
|
|
|
|
i = 1 + digit * skip;
|
1999-01-21 08:11:06 +00:00
|
|
|
if (count <= scan[i].bm_bighint) {
|
|
|
|
|
/*
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* The allocation might fit beginning in the i'th subtree.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
r = blst_meta_alloc(&scan[i], cursor + digit * radix,
|
|
|
|
|
count, radix);
|
1999-01-21 08:11:06 +00:00
|
|
|
if (r != SWAPBLK_NONE) {
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (scan[i].bm_bitmap == 0)
|
|
|
|
|
scan->bm_bitmap ^= bit;
|
2017-06-13 17:49:49 +00:00
|
|
|
return (r);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
}
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
cursor = blk;
|
|
|
|
|
} while ((mask ^= bit) != 0);
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* We couldn't allocate count in this subtree. If the whole tree was
|
|
|
|
|
* scanned, and the last tree node is allocated, update bighint.
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (scan_from_start && !(digit == BLIST_META_RADIX - 1 &&
|
|
|
|
|
scan[i].bm_bighint == BLIST_MAX_ALLOC))
|
1999-01-21 08:11:06 +00:00
|
|
|
scan->bm_bighint = count - 1;
|
2017-06-18 18:23:39 +00:00
|
|
|
|
|
|
|
|
return (SWAPBLK_NONE);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLST_LEAF_FREE() - free allocated block from leaf bitmap
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
static void
|
2017-07-17 23:16:33 +00:00
|
|
|
blst_leaf_free(blmeta_t *scan, daddr_t blk, int count)
|
|
|
|
|
{
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
u_daddr_t mask;
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* free some data in this bitmap
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* mask=0000111111111110000
|
1999-01-21 08:11:06 +00:00
|
|
|
* \_________/\__/
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
* count n
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = bitrange(blk & BLIST_BMAP_MASK, count);
|
|
|
|
|
if (scan->bm_bitmap & mask)
|
Modify blst_leaf_alloc to take only the cursor argument.
Modify blst_leaf_alloc to find allocations that cross the boundary between
one leaf node and the next when those two leaves descend from the same
meta node.
Update the hint field for leaves so that it represents a bound on how
large an allocation can begin in that leaf, where it currently represents
a bound on how large an allocation can be found within the boundaries of
the leaf.
The first phase of blst_leaf_alloc currently shrinks sequences of
consecutive 1-bits in mask until each has been shrunken by count-1 bits,
so that any bits remaining show where an allocation can begin, or until
all the bits have disappeared, in which case the allocation fails. This
change amends that so that the high-order bit is copied, as if, when the
last block was free, it was followed by an endless stream of free
blocks. It also amends the early stopping condition, so that the shrinking
of 1-sequences stops early when there are none, or there is only one
unbounded one remaining.
The search for the first set bit is unchanged, and the code path
thereafter is mostly unchanged unless the first set bit is in a position
that makes some of those copied sign bits matter. In that case, we look
for a next leaf, and at what blocks it can provide, to see if a
cross-boundary allocation is possible.
The hint is updated on a successful allocation that clears the last bit,
but it not updated on a failed allocation that leaves the last bit
set. So, as long as the last block is free, the hint value for the leaf is
large. As long as the last block is free, and there's a next leaf, a large
allocation can begin here, perhaps. A stricter rule than this would mean
that allocations and frees in one leaf could require hint updates to the
preceding leaf, and this change seeks to leave the freeing code
unmodified.
Define BLIST_BMAP_MASK, and use it for bit masking in blst_leaf_free and
blist_leaf_fill, as well as in blst_leaf_alloc.
Correct a panic message in blst_leaf_free.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: markj (an earlier version)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D11819
2017-09-16 18:12:15 +00:00
|
|
|
panic("freeing free block");
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bitmap |= mask;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLST_META_FREE() - free allocated blocks from radix tree meta info
|
|
|
|
|
*
|
|
|
|
|
* This support routine frees a range of blocks from the bitmap.
|
|
|
|
|
* The range must be entirely enclosed by this radix node. If a
|
|
|
|
|
* meta node, we break the range down recursively to free blocks
|
|
|
|
|
* in subnodes (which means that this code can free an arbitrary
|
|
|
|
|
* range whereas the allocation code cannot allocate an arbitrary
|
|
|
|
|
* range).
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
static void
|
2017-08-13 16:39:49 +00:00
|
|
|
blst_meta_free(blmeta_t *scan, daddr_t freeBlk, daddr_t count, u_daddr_t radix)
|
2017-06-27 17:45:26 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t blk, endBlk, i, skip;
|
|
|
|
|
int digit, endDigit;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* We could probably do a better job here. We are required to make
|
|
|
|
|
* bighint at least as large as the biggest allocable block of data.
|
|
|
|
|
* If we just shoehorn it, a little extra overhead will be incurred
|
|
|
|
|
* on the next allocation (but only that one typically).
|
1999-01-21 08:11:06 +00:00
|
|
|
*/
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bighint = BLIST_MAX_ALLOC;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (radix == BLIST_BMAP_RADIX)
|
|
|
|
|
return (blst_leaf_free(scan, freeBlk, count));
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
endBlk = ummin(freeBlk + count, (freeBlk + radix) & -radix);
|
2003-01-11 01:09:51 +00:00
|
|
|
radix /= BLIST_META_RADIX;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
skip = radix_to_skip(radix);
|
|
|
|
|
blk = freeBlk & -radix;
|
|
|
|
|
digit = (blk / radix) & BLIST_META_MASK;
|
|
|
|
|
endDigit = 1 + (((endBlk - 1) / radix) & BLIST_META_MASK);
|
|
|
|
|
scan->bm_bitmap |= bitrange(digit, endDigit - digit);
|
|
|
|
|
for (i = 1 + digit * skip; blk < endBlk; i += skip) {
|
1999-01-21 08:11:06 +00:00
|
|
|
blk += radix;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
count = ummin(blk, endBlk) - freeBlk;
|
|
|
|
|
blst_meta_free(&scan[i], freeBlk, count, radix);
|
|
|
|
|
freeBlk = blk;
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
* BLST_COPY() - copy one radix tree to another
|
1999-01-21 08:11:06 +00:00
|
|
|
*
|
|
|
|
|
* Locates free space in the source tree and frees it in the destination
|
|
|
|
|
* tree. The space may not already be free in the destination.
|
|
|
|
|
*/
|
2017-07-17 23:16:33 +00:00
|
|
|
static void
|
2017-08-01 03:51:26 +00:00
|
|
|
blst_copy(blmeta_t *scan, daddr_t blk, daddr_t radix, blist_t dest,
|
|
|
|
|
daddr_t count)
|
2017-07-17 23:16:33 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t endBlk, i, skip;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Leaf node
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
if (radix == BLIST_BMAP_RADIX) {
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
u_daddr_t v = scan->bm_bitmap;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
if (v == (u_daddr_t)-1) {
|
|
|
|
|
blist_free(dest, blk, count);
|
|
|
|
|
} else if (v != 0) {
|
|
|
|
|
int i;
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
for (i = 0; i < count; ++i) {
|
2017-06-05 17:14:16 +00:00
|
|
|
if (v & ((u_daddr_t)1 << i))
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(dest, blk + i, 1);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Meta node
|
|
|
|
|
*/
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
if (scan->bm_bitmap == 0) {
|
1999-01-21 08:11:06 +00:00
|
|
|
/*
|
|
|
|
|
* Source all allocated, leave dest allocated
|
|
|
|
|
*/
|
|
|
|
|
return;
|
2017-07-17 23:16:33 +00:00
|
|
|
}
|
1999-01-21 08:11:06 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
endBlk = blk + count;
|
2017-08-01 03:51:26 +00:00
|
|
|
radix /= BLIST_META_RADIX;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
skip = radix_to_skip(radix);
|
|
|
|
|
for (i = 1; blk < endBlk; i += skip) {
|
1999-01-21 08:11:06 +00:00
|
|
|
blk += radix;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
count = radix;
|
|
|
|
|
if (blk >= endBlk)
|
|
|
|
|
count -= blk - endBlk;
|
|
|
|
|
blst_copy(&scan[i], blk - radix, radix, dest, count);
|
1999-01-21 08:11:06 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2002-12-15 19:17:57 +00:00
|
|
|
/*
|
|
|
|
|
* BLST_LEAF_FILL() - allocate specific blocks in leaf bitmap
|
|
|
|
|
*
|
|
|
|
|
* This routine allocates all blocks in the specified range
|
|
|
|
|
* regardless of any existing allocations in that range. Returns
|
|
|
|
|
* the number of blocks allocated by the call.
|
|
|
|
|
*/
|
2017-06-09 16:19:24 +00:00
|
|
|
static daddr_t
|
2002-12-15 19:17:57 +00:00
|
|
|
blst_leaf_fill(blmeta_t *scan, daddr_t blk, int count)
|
|
|
|
|
{
|
2017-06-09 16:19:24 +00:00
|
|
|
daddr_t nblks;
|
2017-06-13 17:49:49 +00:00
|
|
|
u_daddr_t mask;
|
2002-12-15 19:17:57 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = bitrange(blk & BLIST_BMAP_MASK, count);
|
2002-12-15 19:17:57 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
/* Count the number of blocks that we are allocating. */
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
nblks = bitcount64(scan->bm_bitmap & mask);
|
2002-12-15 19:17:57 +00:00
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
scan->bm_bitmap &= ~mask;
|
2017-06-13 17:49:49 +00:00
|
|
|
return (nblks);
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* BLIST_META_FILL() - allocate specific blocks at a meta node
|
|
|
|
|
*
|
|
|
|
|
* This routine allocates the specified range of blocks,
|
|
|
|
|
* regardless of any existing allocations in the range. The
|
|
|
|
|
* range must be within the extent of this node. Returns the
|
|
|
|
|
* number of blocks allocated by the call.
|
|
|
|
|
*/
|
2017-06-09 16:19:24 +00:00
|
|
|
static daddr_t
|
2017-08-13 16:39:49 +00:00
|
|
|
blst_meta_fill(blmeta_t *scan, daddr_t allocBlk, daddr_t count, u_daddr_t radix)
|
2017-06-27 17:45:26 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t blk, endBlk, i, nblks, skip;
|
|
|
|
|
int digit;
|
2002-12-15 19:17:57 +00:00
|
|
|
|
2017-07-24 17:23:53 +00:00
|
|
|
if (radix == BLIST_BMAP_RADIX)
|
|
|
|
|
return (blst_leaf_fill(scan, allocBlk, count));
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
|
|
|
|
|
endBlk = ummin(allocBlk + count, (allocBlk + radix) & -radix);
|
|
|
|
|
radix /= BLIST_META_RADIX;
|
2017-08-01 03:51:26 +00:00
|
|
|
skip = radix_to_skip(radix);
|
2017-08-13 16:39:49 +00:00
|
|
|
blk = allocBlk & -radix;
|
2017-06-27 17:45:26 +00:00
|
|
|
nblks = 0;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
while (blk < endBlk) {
|
|
|
|
|
digit = (blk / radix) & BLIST_META_MASK;
|
|
|
|
|
i = 1 + digit * skip;
|
2002-12-15 19:17:57 +00:00
|
|
|
blk += radix;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
count = ummin(blk, endBlk) - allocBlk;
|
|
|
|
|
nblks += blst_meta_fill(&scan[i], allocBlk, count, radix);
|
|
|
|
|
if (scan[i].bm_bitmap == 0)
|
|
|
|
|
scan->bm_bitmap &= ~((u_daddr_t)1 << digit);
|
|
|
|
|
allocBlk = blk;
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
2017-07-24 17:23:53 +00:00
|
|
|
return (nblks);
|
2002-12-15 19:17:57 +00:00
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:11:06 +00:00
|
|
|
#ifdef BLIST_DEBUG
|
|
|
|
|
|
2017-07-17 23:16:33 +00:00
|
|
|
static void
|
2017-08-01 03:51:26 +00:00
|
|
|
blst_radix_print(blmeta_t *scan, daddr_t blk, daddr_t radix, int tab)
|
1999-01-21 08:11:06 +00:00
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
daddr_t skip;
|
|
|
|
|
u_daddr_t bit, mask;
|
|
|
|
|
int digit;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
if (radix == BLIST_BMAP_RADIX) {
|
|
|
|
|
printf(
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
"%*.*s(%08llx,%lld): bitmap %0*llx big=%lld\n",
|
1999-01-21 08:11:06 +00:00
|
|
|
tab, tab, "",
|
2002-12-15 19:17:57 +00:00
|
|
|
(long long)blk, (long long)radix,
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
1 + (BLIST_BMAP_RADIX - 1) / 4,
|
|
|
|
|
(long long)scan->bm_bitmap,
|
2002-12-15 19:17:57 +00:00
|
|
|
(long long)scan->bm_bighint
|
1999-01-21 08:11:06 +00:00
|
|
|
);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
printf(
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
"%*.*s(%08llx): subtree (%lld/%lld) bitmap %0*llx big=%lld {\n",
|
1999-01-21 08:11:06 +00:00
|
|
|
tab, tab, "",
|
2002-12-15 19:17:57 +00:00
|
|
|
(long long)blk, (long long)radix,
|
|
|
|
|
(long long)radix,
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
1 + (BLIST_META_RADIX - 1) / 4,
|
|
|
|
|
(long long)scan->bm_bitmap,
|
2002-12-15 19:17:57 +00:00
|
|
|
(long long)scan->bm_bighint
|
1999-01-21 08:11:06 +00:00
|
|
|
);
|
|
|
|
|
|
2017-08-01 03:51:26 +00:00
|
|
|
radix /= BLIST_META_RADIX;
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
skip = radix_to_skip(radix);
|
1999-01-21 08:11:06 +00:00
|
|
|
tab += 4;
|
|
|
|
|
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
mask = scan->bm_bitmap;
|
|
|
|
|
/* Examine the nonempty subtree associated with each bit set in mask */
|
|
|
|
|
do {
|
|
|
|
|
bit = mask & -mask;
|
|
|
|
|
digit = bitpos(bit);
|
|
|
|
|
blst_radix_print(&scan[1 + digit * skip], blk + digit * radix,
|
|
|
|
|
radix, tab);
|
|
|
|
|
} while ((mask ^= bit) != 0);
|
1999-01-21 08:11:06 +00:00
|
|
|
tab -= 4;
|
|
|
|
|
|
|
|
|
|
printf(
|
|
|
|
|
"%*.*s}\n",
|
|
|
|
|
tab, tab, ""
|
|
|
|
|
);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifdef BLIST_DEBUG
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
main(int ac, char **av)
|
|
|
|
|
{
|
Allow allocations across meta boundaries.
Remove restrictions that prevent allocation requests to cross the
boundary between two meta nodes.
Replace the bmu_avail field in meta nodes with a bitmap that identifies
which subtrees have some free memory, and iterate over the nonempty
subtrees only in blst_meta_alloc. If free memory is scarce, this should
make searching for it faster.
Put the code for handling the next-leaf allocation in a separate
function. When taking blocks from the next leaf empties the leaf, be
sure to clear the appropriate bit in its parent, and so on, up to the
least-common ancestor of this leaf and the next.
Eliminate special terminator nodes, and rely instead on the fact that
there is a 0-bit at the end of the bitmask at the root of the tree that
will stop a meta_alloc search, or a next-leaf search, before the search
falls off the end of the tree. Make sure that the tree is big enough to
have space for that 0-bit.
Eliminate special all-free indicators. Lazy initialization of subtrees
stands in the way of having an allocation span a meta-node boundary, so
a subtree of all free blocks is not treated specially. Subtrees of
all-allocated blocks are still recognized by looking at the bitmask at
the root and finding 0.
Don't print all-allocated subtrees. Do print the bitmasks for meta
nodes, when tree-printing.
Submitted by: Doug Moore <dougm@rice.edu>
Reviewed by: alc
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D12635
2018-11-13 18:40:01 +00:00
|
|
|
int size = BLIST_META_RADIX * BLIST_BMAP_RADIX;
|
1999-01-21 08:11:06 +00:00
|
|
|
int i;
|
|
|
|
|
blist_t bl;
|
2017-09-10 17:46:03 +00:00
|
|
|
struct sbuf *s;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
|
|
|
|
for (i = 1; i < ac; ++i) {
|
|
|
|
|
const char *ptr = av[i];
|
|
|
|
|
if (*ptr != '-') {
|
|
|
|
|
size = strtol(ptr, NULL, 0);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
ptr += 2;
|
|
|
|
|
fprintf(stderr, "Bad option: %s\n", ptr - 2);
|
|
|
|
|
exit(1);
|
|
|
|
|
}
|
2008-05-05 19:48:54 +00:00
|
|
|
bl = blist_create(size, M_WAITOK);
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(bl, 0, size);
|
|
|
|
|
|
|
|
|
|
for (;;) {
|
|
|
|
|
char buf[1024];
|
2017-06-07 16:04:34 +00:00
|
|
|
long long da = 0;
|
|
|
|
|
long long count = 0;
|
1999-01-21 08:11:06 +00:00
|
|
|
|
2017-06-13 17:49:49 +00:00
|
|
|
printf("%lld/%lld/%lld> ", (long long)blist_avail(bl),
|
2002-12-15 19:17:57 +00:00
|
|
|
(long long)size, (long long)bl->bl_radix);
|
1999-01-21 08:11:06 +00:00
|
|
|
fflush(stdout);
|
|
|
|
|
if (fgets(buf, sizeof(buf), stdin) == NULL)
|
|
|
|
|
break;
|
|
|
|
|
switch(buf[0]) {
|
|
|
|
|
case 'r':
|
2002-12-15 19:17:57 +00:00
|
|
|
if (sscanf(buf + 1, "%lld", &count) == 1) {
|
2017-06-07 16:04:34 +00:00
|
|
|
blist_resize(&bl, count, 1, M_WAITOK);
|
1999-01-21 08:11:06 +00:00
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
case 'p':
|
|
|
|
|
blist_print(bl);
|
|
|
|
|
break;
|
2017-09-10 17:46:03 +00:00
|
|
|
case 's':
|
|
|
|
|
s = sbuf_new_auto();
|
|
|
|
|
blist_stats(bl, s);
|
|
|
|
|
sbuf_finish(s);
|
|
|
|
|
printf("%s", sbuf_data(s));
|
|
|
|
|
sbuf_delete(s);
|
|
|
|
|
break;
|
1999-01-21 08:11:06 +00:00
|
|
|
case 'a':
|
2002-12-15 19:17:57 +00:00
|
|
|
if (sscanf(buf + 1, "%lld", &count) == 1) {
|
1999-01-21 08:11:06 +00:00
|
|
|
daddr_t blk = blist_alloc(bl, count);
|
2002-12-15 19:17:57 +00:00
|
|
|
printf(" R=%08llx\n", (long long)blk);
|
1999-01-21 08:11:06 +00:00
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
case 'f':
|
2017-06-07 16:04:34 +00:00
|
|
|
if (sscanf(buf + 1, "%llx %lld", &da, &count) == 2) {
|
1999-01-21 08:11:06 +00:00
|
|
|
blist_free(bl, da, count);
|
|
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
break;
|
2002-12-15 19:17:57 +00:00
|
|
|
case 'l':
|
2017-06-07 16:04:34 +00:00
|
|
|
if (sscanf(buf + 1, "%llx %lld", &da, &count) == 2) {
|
2017-06-09 16:19:24 +00:00
|
|
|
printf(" n=%jd\n",
|
|
|
|
|
(intmax_t)blist_fill(bl, da, count));
|
2002-12-15 19:17:57 +00:00
|
|
|
} else {
|
|
|
|
|
printf("?\n");
|
|
|
|
|
}
|
|
|
|
|
break;
|
1999-01-21 08:11:06 +00:00
|
|
|
case '?':
|
|
|
|
|
case 'h':
|
|
|
|
|
puts(
|
|
|
|
|
"p -print\n"
|
2017-09-10 17:46:03 +00:00
|
|
|
"s -stats\n"
|
1999-01-21 08:11:06 +00:00
|
|
|
"a %d -allocate\n"
|
|
|
|
|
"f %x %d -free\n"
|
2002-12-15 19:17:57 +00:00
|
|
|
"l %x %d -fill\n"
|
1999-01-21 08:11:06 +00:00
|
|
|
"r %d -resize\n"
|
|
|
|
|
"h/? -help"
|
|
|
|
|
);
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
printf("?\n");
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return(0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
panic(const char *ctl, ...)
|
|
|
|
|
{
|
|
|
|
|
va_list va;
|
|
|
|
|
|
|
|
|
|
va_start(va, ctl);
|
|
|
|
|
vfprintf(stderr, ctl, va);
|
|
|
|
|
fprintf(stderr, "\n");
|
|
|
|
|
va_end(va);
|
|
|
|
|
exit(1);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif
|
