d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
f9319cf885
freebsd-skq/sys/vm/vm_radix.c

891 lines
23 KiB
C
Raw Normal View History

Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Copyright (c) 2011 Jeffrey Roberson <jeff@freebsd.org>
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
* Copyright (c) 2008 Mayur Shardul <mayur.shardul@gmail.com>
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
*/
/*
* Radix tree implementation.
*/
#include <sys/cdefs.h>
#include <sys/param.h>
#include <sys/conf.h>
#include <sys/systm.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/queue.h>
#include <sys/param.h>
#include <sys/lock.h>
#include <sys/mutex.h>
#include <sys/ktr.h>
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
#include <vm/uma.h>
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
#include <vm/vm.h>
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
#include <vm/vm_param.h>
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
#include <vm/vm_extern.h>
#include <vm/vm_kern.h>
#include <vm/vm_page.h>
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
#ifndef UMA_MD_SMALL_ALLOC
#include <vm/vm_map.h>
#endif
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
#include <vm/vm_radix.h>
#include <vm/vm_object.h>
#include <sys/kdb.h>
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
#ifndef UMA_MD_SMALL_ALLOC
#define VM_RADIX_RNODE_MAP_SCALE (1024 * 1024 / 2)
#define VM_RADIX_WIDTH 4
/*
* Bits of height in root.
* The mask of smaller power of 2 containing VM_RADIX_LIMIT.
*/
#define VM_RADIX_HEIGHT 0x1f
#else
- Exclude vm_radix_shrink() from the interface but retain the code still as it can be useful. - Make most of the interface private as it is unnecessary public right now. This will help in making nodes changing with arch and still avoid namespace pollution.
2012-03-01 00:54:08 +00:00
#define VM_RADIX_WIDTH 5
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
/* See the comment above. */
#define VM_RADIX_HEIGHT 0xf
#endif
- Exclude vm_radix_shrink() from the interface but retain the code still as it can be useful. - Make most of the interface private as it is unnecessary public right now. This will help in making nodes changing with arch and still avoid namespace pollution.
2012-03-01 00:54:08 +00:00
#define VM_RADIX_COUNT (1 << VM_RADIX_WIDTH)
#define VM_RADIX_MASK (VM_RADIX_COUNT - 1)
#define VM_RADIX_MAXVAL ((vm_pindex_t)-1)
#define VM_RADIX_LIMIT howmany((sizeof(vm_pindex_t) * NBBY), VM_RADIX_WIDTH)
/* Flag bits stored in node pointers. */
#define VM_RADIX_FLAGS 0x3
/* Calculates maximum value for a tree of height h. */
#define VM_RADIX_MAX(h) \
((h) == VM_RADIX_LIMIT ? VM_RADIX_MAXVAL : \
(((vm_pindex_t)1 << ((h) * VM_RADIX_WIDTH)) - 1))
CTASSERT(VM_RADIX_HEIGHT >= VM_RADIX_LIMIT);
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
CTASSERT((sizeof(u_int) * NBBY) >= VM_RADIX_LIMIT);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
- Exclude vm_radix_shrink() from the interface but retain the code still as it can be useful. - Make most of the interface private as it is unnecessary public right now. This will help in making nodes changing with arch and still avoid namespace pollution.
2012-03-01 00:54:08 +00:00
struct vm_radix_node {
void *rn_child[VM_RADIX_COUNT]; /* Child nodes. */
volatile uint32_t rn_count; /* Valid children. */
};
Fix a compile time bug by adding a check just after the struct definition
2012-03-06 23:37:53 +00:00
CTASSERT(sizeof(struct vm_radix_node) < PAGE_SIZE);
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
static uma_zone_t vm_radix_node_zone;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
#ifndef UMA_MD_SMALL_ALLOC
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
static vm_map_t rnode_map;
static u_long rnode_map_scale;
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
static void *
vm_radix_node_zone_allocf(uma_zone_t zone, int size, uint8_t *flags, int wait)
{
vm_offset_t addr;
vm_page_t m;
int pflags;
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
/* Inform UMA that this allocator uses rnode_map. */
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
*flags = UMA_SLAB_KERNEL;
pflags = VM_ALLOC_WIRED | VM_ALLOC_NOOBJ;
/*
* As kmem_alloc_nofault() can however fail, let just assume that
* M_NOWAIT is on and act accordingly.
*/
pflags |= ((wait & M_USE_RESERVE) != 0) ? VM_ALLOC_INTERRUPT :
VM_ALLOC_SYSTEM;
if ((wait & M_ZERO) != 0)
pflags |= VM_ALLOC_ZERO;
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
addr = kmem_alloc_nofault(rnode_map, size);
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
if (addr == 0)
return (NULL);
/* Just one page allocation is assumed here. */
m = vm_page_alloc(NULL, OFF_TO_IDX(addr - VM_MIN_KERNEL_ADDRESS),
pflags);
if (m == NULL) {
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
kmem_free(rnode_map, addr, size);
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
return (NULL);
}
if ((wait & M_ZERO) != 0 && (m->flags & PG_ZERO) == 0)
pmap_zero_page(m);
pmap_qenter(addr, &m, 1);
return ((void *)addr);
}
static void
vm_radix_node_zone_freef(void *item, int size, uint8_t flags)
{
vm_page_t m;
vm_offset_t voitem;
MPASS((flags & UMA_SLAB_KERNEL) != 0);
/* Just one page allocation is assumed here. */
voitem = (vm_offset_t)item;
m = PHYS_TO_VM_PAGE(pmap_kextract(voitem));
pmap_qremove(voitem, 1);
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
vm_page_lock(m);
vm_page_unwire(m, 0);
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
vm_page_free(m);
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
vm_page_unlock(m);
kmem_free(rnode_map, voitem, size);
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
}
static void
init_vm_radix_alloc(void *dummy __unused)
{
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
uma_zone_set_max(vm_radix_node_zone, rnode_map_scale);
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
uma_zone_set_allocf(vm_radix_node_zone, vm_radix_node_zone_allocf);
uma_zone_set_freef(vm_radix_node_zone, vm_radix_node_zone_freef);
}
SYSINIT(vm_radix, SI_SUB_KMEM, SI_ORDER_SECOND, init_vm_radix_alloc, NULL);
#endif
/*
* Radix node zone destructor.
*/
#ifdef INVARIANTS
static void
vm_radix_node_zone_dtor(void *mem, int size, void *arg)
{
struct vm_radix_node *rnode;
rnode = mem;
KASSERT(rnode->rn_count == 0,
("vm_radix_node_put: Freeing a node with %d children\n",
rnode->rn_count));
}
#endif
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Allocate a radix node. Initializes all elements to 0.
*/
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
static __inline struct vm_radix_node *
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
vm_radix_node_get(void)
{
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
return (uma_zalloc(vm_radix_node_zone, M_NOWAIT | M_ZERO));
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
/*
* Free radix node.
*/
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
static __inline void
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
vm_radix_node_put(struct vm_radix_node *rnode)
{
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
uma_zfree(vm_radix_node_zone, rnode);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
/*
* Return the position in the array for a given level.
*/
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
static __inline int
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
vm_radix_slot(vm_pindex_t index, int level)
{
return ((index >> (level * VM_RADIX_WIDTH)) & VM_RADIX_MASK);
}
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
/*
* Initialize the radix node submap (for architectures not supporting
* direct-mapping) and the radix node zone.
*
* WITNESS reports a lock order reversal, for architectures not
* supporting direct-mapping, between the "system map" lock
* and the "vm object" lock. This is because the well established ordering
* "system map" -> "vm object" is not honoured in this case as allocating
* from the radix node submap ends up adding a mapping entry to it, meaning
* it is necessary to lock the submap. However, the radix node submap is
* a leaf and self-contained, thus a deadlock cannot happen here and
* adding MTX_NOWITNESS to all map locks would be largerly sub-optimal.
*/
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
void
vm_radix_init(void)
{
Fix the nodes allocator in architectures without direct-mapping: - Fix bugs in the free path where the pages were not unwired and relevant locking wasn't acquired. - Introduce the rnode_map, submap of kernel_map, where to allocate from. The reason is that, in architectures without direct-mapping, kmem_alloc*() will try to insert the newly created mapping while holding the vm_object lock introducing a LOR or lock recursion. rnode_map is however a leafly-used submap, thus there cannot be any deadlock. Notes: Size the submap in order to be, by default, around 64 MB and decrase the size of the nodes as the allocation will be much smaller (and when the compacting code in the vm_radix will be implemented this will aim for much less space to be used). However note that the size of the submap can be changed at boot time via the hw.rnode_map_scale scaling factor. - Use uma_zone_set_max() covering the size of the submap. Tested by: flo
2012-03-16 15:41:07 +00:00
#ifndef UMA_MD_SMALL_ALLOC
vm_offset_t maxaddr, minaddr;
rnode_map_scale = VM_RADIX_RNODE_MAP_SCALE;
TUNABLE_ULONG_FETCH("hw.rnode_map_scale", &rnode_map_scale);
rnode_map = kmem_suballoc(kernel_map, &minaddr, &maxaddr,
rnode_map_scale * sizeof(struct vm_radix_node), FALSE);
rnode_map->system_map = 1;
#endif
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
vm_radix_node_zone = uma_zcreate("RADIX NODE",
sizeof(struct vm_radix_node), NULL,
#ifdef INVARIANTS
vm_radix_node_zone_dtor,
#else
NULL,
#endif
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
NULL, NULL, VM_RADIX_HEIGHT, UMA_ZONE_VM);
}
/*
* Extract the root node and height from a radix tree with a single load.
*/
static __inline int
vm_radix_height(struct vm_radix *rtree, struct vm_radix_node **rnode)
{
uintptr_t root;
int height;
root = rtree->rt_root;
height = root & VM_RADIX_HEIGHT;
*rnode = (struct vm_radix_node *)(root - height);
return (height);
}
/*
* Set the root node and height for a radix tree.
*/
static inline void
vm_radix_setroot(struct vm_radix *rtree, struct vm_radix_node *rnode,
int height)
{
uintptr_t root;
root = (uintptr_t)rnode | height;
rtree->rt_root = root;
Use an UMA zone for the radix node. This avoids the problem to check for the kernel_map/kmem_map recursion because it uses direct mapping provided by amd64 to avoid object and map search and recursion. Probabilly all the others architectures using UMA_MD_SMALL_ALLOC are also fixed by this, but other remains, where the most notable case is i386. For it a solution has still to be determined. A way to do this would be to have a reserved map just for radix node and mark all accesses to its lock to be witness safe, but that would still be unoptimal due to the large amount of virtual address space needed to cater the whole tree.
2011-10-28 01:56:36 +00:00
}
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
static inline void
vm_radix_unwind_heightup(struct vm_radix *rtree, struct vm_radix_node *root,
struct vm_radix_node *iroot, int ilevel)
{
struct vm_radix_node *rnode;
CTR4(KTR_VM, "unwind: tree %p, root %p, iroot %p, ilevel %d",
rtree, root, iroot, ilevel);
while (iroot != root && root != NULL) {
rnode = root;
MPASS(rnode->rn_count == 0 || rnode->rn_count == 1);
rnode->rn_count = 0;
root = rnode->rn_child[0];
vm_radix_node_put(rnode);
}
vm_radix_setroot(rtree, iroot, ilevel);
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
static inline void *
vm_radix_match(void *child, int color)
{
uintptr_t c;
c = (uintptr_t)child;
if ((c & color) == 0)
return (NULL);
return ((void *)(c & ~VM_RADIX_FLAGS));
}
vm_object_terminate() doesn't actually free the pages in the splay tree. Reclaim all the nodes related to the radix tree for a specified vm_object when calling vm_object_terminate() via the newly added interface vm_radix_reclaim_nodes(). The function is recursive, but we have a well-defined maximum depth, thus the amount of necessary stack can be easilly calculated. Reported by: alc Discussed and reviewed by: jeff
2011-11-01 03:40:38 +00:00
static void
vm_radix_reclaim_allnodes_internal(struct vm_radix_node *rnode, int level)
{
int slot;
MPASS(rnode != NULL && level >= 0);
/*
* Level 0 just contains pages as children, thus make it a special
* case, free the node and return.
*/
if (level == 0) {
CTR2(KTR_VM, "reclaiming: node %p, level %d", rnode, level);
rnode->rn_count = 0;
vm_radix_node_put(rnode);
return;
}
for (slot = 0; slot < VM_RADIX_COUNT && rnode->rn_count != 0; slot++) {
if (rnode->rn_child[slot] == NULL)
continue;
CTR3(KTR_VM,
"reclaiming: node %p, level %d recursing in slot %d",
rnode, level, slot);
vm_radix_reclaim_allnodes_internal(rnode->rn_child[slot],
level - 1);
rnode->rn_count--;
}
MPASS(rnode->rn_count == 0);
CTR2(KTR_VM, "reclaiming: node %p, level %d", rnode, level);
vm_radix_node_put(rnode);
}
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Inserts the key-value pair in to the radix tree. Returns errno.
* Panics if the key already exists.
*/
int
vm_radix_insert(struct vm_radix *rtree, vm_pindex_t index, void *val)
{
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
struct vm_radix_node *iroot, *rnode, *root;
u_int allocmsk;
int clev, ilevel, level, slot;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
CTR3(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"insert: tree %p, index %ju, val %p", rtree, (uintmax_t)index, val);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
if (index == -1)
panic("vm_radix_insert: -1 is not a valid index.\n");
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level = vm_radix_height(rtree, &root);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Increase the height by adding nodes at the root until
* there is sufficient space.
*/
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
ilevel = level;
iroot = root;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
while (level == 0 || index > VM_RADIX_MAX(level)) {
Fix format string for the pindex members as they should be treated as uintmax_t for compatibility among 32/64 bits.
2012-01-29 16:29:06 +00:00
CTR3(KTR_VM, "insert: expanding %ju > %ju height %d",
(uintmax_t)index, (uintmax_t)VM_RADIX_MAX(level), level);
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level++;
KASSERT(level <= VM_RADIX_LIMIT,
("vm_radix_insert: Tree %p height %d too tall",
rtree, level));
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Only allocate tree nodes if they are needed.
*/
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
if (root == NULL || root->rn_count != 0) {
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
rnode = vm_radix_node_get();
Add more KTR points for failure in vm_radix_insert().
2011-11-20 14:51:27 +00:00
if (rnode == NULL) {
CTR4(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"insert: tree %p, root %p, index: %ju, level: %d ENOMEM",
rtree, root, (uintmax_t)index, level);
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
vm_radix_unwind_heightup(rtree, root, iroot,
ilevel);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
return (ENOMEM);
Add more KTR points for failure in vm_radix_insert().
2011-11-20 14:51:27 +00:00
}
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
/*
* Store the new pointer with a memory barrier so
* that it is visible before the new root.
*/
if (root) {
atomic_store_rel_ptr((volatile uintptr_t *)
&rnode->rn_child[0], (uintptr_t)root);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
rnode->rn_count = 1;
}
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
root = rnode;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
vm_radix_setroot(rtree, root, level);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
/* Now that the tree is tall enough, fill in the path to the index. */
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
allocmsk = 0;
clev = level;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
rnode = root;
for (level = level - 1; level > 0; level--) {
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
slot = vm_radix_slot(index, level);
/* Add the required intermidiate nodes. */
if (rnode->rn_child[slot] == NULL) {
rnode->rn_child[slot] = vm_radix_node_get();
Add more KTR points for failure in vm_radix_insert().
2011-11-20 14:51:27 +00:00
if (rnode->rn_child[slot] == NULL) {
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR5(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"insert: tree %p, index %ju, level %d, slot %d, rnode %p ENOMEM",
rtree, (uintmax_t)index, level, slot,
rnode);
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR4(KTR_VM,
"insert: tree %p, rnode %p, child %p, count %u ENOMEM",
rtree, rnode, rnode->rn_child[slot],
rnode->rn_count);
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
MPASS(level != clev || allocmsk == 0);
while (allocmsk != 0) {
rnode = root;
level = clev;
level--;
fix KTR consistency I'm committing this on behalf of Attilio as he cannot access svn right now.
2012-02-05 18:55:20 +00:00
slot = vm_radix_slot(index, level);
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
CTR4(KTR_VM,
"insert: unwind root %p, level %d, slot %d, allocmsk: 0x%x",
root, level, slot, allocmsk);
MPASS(level >= (ffs(allocmsk) - 1));
while (level > (ffs(allocmsk) - 1)) {
MPASS(level > 0);
slot = vm_radix_slot(index,
level);
rnode = rnode->rn_child[slot];
level--;
}
MPASS((allocmsk & (1 << level)) != 0);
allocmsk &= ~(1 << level);
rnode->rn_count--;
vm_radix_node_put(rnode->rn_child[slot]);
rnode->rn_child[slot] = NULL;
}
vm_radix_unwind_heightup(rtree, root, iroot,
ilevel);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
return (ENOMEM);
Add more KTR points for failure in vm_radix_insert().
2011-11-20 14:51:27 +00:00
}
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
rnode->rn_count++;
Remove the panic from vm_radix_insert() and propagate the error to the callers of vm_page_insert(). The default action for every caller is to unwind-back the operation besides vm_page_rename() where this has proven to be impossible to do. For that case, it just spins until the page is not available to be allocated. However, due to vm_page_rename() to be mostly rare (and having never hit this panic in the past) it is tought to be a very seldom thing and not a possible performance factor. The patch has been tested with an atomic counter returning NULL from the zone allocator every 1/100000 allocations. Per-printf, I've verified that a typical buildkernel could trigger this 30 times. The patch survived to 2 hours of repeated buildkernel/world. Several technical notes: - The vm_page_insert() is moved, in several callers, closer to failure points. This could be committed separately before vmcontention hits the tree just to verify -CURRENT is happy with it. - vm_page_rename() does not need to have the page lock in the callers as it hide that as an implementation detail. Do the locking internally. - now vm_page_insert() returns an int, with 0 meaning everything was ok, thus KPI is broken by this patch.
2012-02-05 17:37:26 +00:00
allocmsk |= (1 << level);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR5(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"insert: tree %p, index %ju, level %d, slot %d, rnode %p",
rtree, (uintmax_t)index, level, slot, rnode);
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR4(KTR_VM,
"insert: tree %p, rnode %p, child %p, count %u",
rtree, rnode, rnode->rn_child[slot], rnode->rn_count);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
rnode = rnode->rn_child[slot];
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
slot = vm_radix_slot(index, 0);
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
MPASS(rnode != NULL);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
KASSERT(rnode->rn_child[slot] == NULL,
("vm_radix_insert: Duplicate value %p at index: %lu\n",
rnode->rn_child[slot], (u_long)index));
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
val = (void *)((uintptr_t)val | VM_RADIX_BLACK);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
rnode->rn_child[slot] = val;
Use atomics for rn_count on leaf node because RED operations happen without the VM_OBJECT_LOCK held, thus can be concurrent with BLACK ones. However, also use a write memory barrier in order to not reorder the operation of decrementing rn_count in respect fetching the pointer. Discussed with: jeff
2011-12-06 22:57:48 +00:00
atomic_add_32(&rnode->rn_count, 1);
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR6(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"insert: tree %p, index %ju, level %d, slot %d, rnode %p, count %u",
rtree, (uintmax_t)index, level, slot, rnode, rnode->rn_count);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
return 0;
}
/*
* Returns the value stored at the index. If the index is not present
* NULL is returned.
*/
void *
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
vm_radix_lookup(struct vm_radix *rtree, vm_pindex_t index, int color)
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
{
struct vm_radix_node *rnode;
int slot;
int level;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level = vm_radix_height(rtree, &rnode);
if (index > VM_RADIX_MAX(level))
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
return NULL;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level--;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
while (rnode) {
slot = vm_radix_slot(index, level);
Fix a bug when the 'rnode' pointer can be NULL and we try to track the children. This helps in debugging case. Reported by: flo
2011-11-26 14:26:37 +00:00
CTR6(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"lookup: tree %p, index %ju, level %d, slot %d, rnode %p, child %p",
rtree, (uintmax_t)index, level, slot, rnode,
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
rnode->rn_child[slot]);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
if (level == 0)
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
return vm_radix_match(rnode->rn_child[slot], color);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
rnode = rnode->rn_child[slot];
level--;
}
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
CTR2(KTR_VM, "lookup: tree %p, index %ju failed", rtree,
(uintmax_t)index);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
return NULL;
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
void *
vm_radix_color(struct vm_radix *rtree, vm_pindex_t index, int color)
{
struct vm_radix_node *rnode;
uintptr_t child;
int slot;
int level;
level = vm_radix_height(rtree, &rnode);
if (index > VM_RADIX_MAX(level))
return NULL;
level--;
while (rnode) {
slot = vm_radix_slot(index, level);
Fix a bug when the 'rnode' pointer can be NULL and we try to track the children. This helps in debugging case. Reported by: flo
2011-11-26 14:26:37 +00:00
CTR6(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"color: tree %p, index %ju, level %d, slot %d, rnode %p, child %p",
rtree, (uintmax_t)index, level, slot, rnode,
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
rnode->rn_child[slot]);
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
if (level == 0)
break;
rnode = rnode->rn_child[slot];
level--;
}
if (rnode == NULL || rnode->rn_child[slot] == NULL)
return (NULL);
child = (uintptr_t)rnode->rn_child[slot];
child &= ~VM_RADIX_FLAGS;
rnode->rn_child[slot] = (void *)(child | color);
return (void *)child;
}
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
* Find the first leaf with a valid node between *startp and end. Return
* the index of the first valid item in the leaf in *startp.
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
*/
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
static struct vm_radix_node *
vm_radix_leaf(struct vm_radix *rtree, vm_pindex_t *startp, vm_pindex_t end)
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
{
struct vm_radix_node *rnode;
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
vm_pindex_t start;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
vm_pindex_t inc;
int slot;
int level;
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
start = *startp;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
restart:
level = vm_radix_height(rtree, &rnode);
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
if (start > VM_RADIX_MAX(level) || (end && start >= end)) {
rnode = NULL;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
goto out;
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
}
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Search the tree from the top for any leaf node holding an index
* between start and end.
*/
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
for (level--; level; level--) {
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
slot = vm_radix_slot(start, level);
Fix a bug when the 'rnode' pointer can be NULL and we try to track the children. This helps in debugging case. Reported by: flo
2011-11-26 14:26:37 +00:00
CTR6(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"leaf: tree %p, index %ju, level %d, slot %d, rnode %p, child %p",
rtree, (uintmax_t)start, level, slot, rnode,
Fix a bug when the 'rnode' pointer can be NULL and we try to track the children. This helps in debugging case. Reported by: flo
2011-11-26 14:26:37 +00:00
(rnode != NULL) ? rnode->rn_child[slot] : NULL);
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
if (rnode->rn_child[slot] != NULL) {
rnode = rnode->rn_child[slot];
continue;
}
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
* Calculate how much to increment our index by
* based on the tree level. We must truncate the
* lower bits to start from the begnning of the
* next leaf.
*/
inc = 1LL << (level * VM_RADIX_WIDTH);
start &= ~VM_RADIX_MAX(level);
Fix a bug in vm_radix_leaf() where the shifting start address can wrap-up at some point. This bug is triggered very easilly by indirect blocks in UFS which grow negative resulting in very high counts. In collabouration with: flo
2012-01-29 16:44:21 +00:00
/* Avoid start address wrapping up. */
if ((VM_RADIX_MAXVAL - start) < inc) {
rnode = NULL;
goto out;
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
start += inc;
slot++;
CTR5(KTR_VM,
Fix format string for the pindex members as they should be treated as uintmax_t for compatibility among 32/64 bits.
2012-01-29 16:29:06 +00:00
"leaf: start %ju end %ju inc %ju mask 0x%jX slot %d",
(uintmax_t)start, (uintmax_t)end, (uintmax_t)inc,
(uintmax_t)~VM_RADIX_MAX(level), slot);
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
for (; slot < VM_RADIX_COUNT; slot++, start += inc) {
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
if (end != 0 && start >= end) {
rnode = NULL;
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
goto out;
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
if (rnode->rn_child[slot]) {
rnode = rnode->rn_child[slot];
break;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
Fix a bug in vm_radix_leaf() where the shifting start address can wrap-up at some point. This bug is triggered very easilly by indirect blocks in UFS which grow negative resulting in very high counts. In collabouration with: flo
2012-01-29 16:44:21 +00:00
if ((VM_RADIX_MAXVAL - start) < inc) {
rnode = NULL;
goto out;
}
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
if (slot == VM_RADIX_COUNT)
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
goto restart;
}
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
out:
*startp = start;
return (rnode);
}
/*
* Looks up as many as cnt values between start and end, and stores
* them in the caller allocated array out. The next index can be used
* to restart the scan. This optimizes forward scans in the tree.
*/
int
vm_radix_lookupn(struct vm_radix *rtree, vm_pindex_t start,
vm_pindex_t end, int color, void **out, int cnt, vm_pindex_t *next)
{
struct vm_radix_node *rnode;
void *val;
int slot;
int outidx;
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
CTR3(KTR_VM, "lookupn: tree %p, start %ju, end %ju",
rtree, (uintmax_t)start, (uintmax_t)end);
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
if (rtree->rt_root == 0)
return (0);
outidx = 0;
while ((rnode = vm_radix_leaf(rtree, &start, end)) != NULL) {
slot = vm_radix_slot(start, 0);
for (; slot < VM_RADIX_COUNT; slot++, start++) {
if (end != 0 && start >= end)
goto out;
val = vm_radix_match(rnode->rn_child[slot], color);
if (val == NULL)
continue;
CTR4(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"lookupn: tree %p index %ju slot %d found child %p",
rtree, (uintmax_t)start, slot, val);
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
out[outidx] = val;
if (++outidx == cnt)
goto out;
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
if (end != 0 && start >= end)
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
break;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
out:
*next = start;
return (outidx);
}
- Extract part of vm_radix_lookupn() into a function that just finds the first leaf page in a specified range. This permits us to make many search & operate functions without much code duplication. - Make a generic iterator for radix items.
2011-10-30 22:57:42 +00:00
void
vm_radix_foreach(struct vm_radix *rtree, vm_pindex_t start, vm_pindex_t end,
int color, void (*iter)(void *))
{
struct vm_radix_node *rnode;
void *val;
int slot;
if (rtree->rt_root == 0)
return;
while ((rnode = vm_radix_leaf(rtree, &start, end)) != NULL) {
slot = vm_radix_slot(start, 0);
for (; slot < VM_RADIX_COUNT; slot++, start++) {
if (end != 0 && start >= end)
return;
val = vm_radix_match(rnode->rn_child[slot], color);
if (val)
iter(val);
}
if (end != 0 && start >= end)
return;
}
}
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
* Look up any entry at a position less than or equal to index.
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
*/
void *
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
vm_radix_lookup_le(struct vm_radix *rtree, vm_pindex_t index, int color)
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
{
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
struct vm_radix_node *rnode;
struct vm_radix_node *child;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
vm_pindex_t max;
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
vm_pindex_t inc;
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
void *val;
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
int slot;
int level;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
CTR2(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"lookup_le: tree %p, index %ju", rtree, (uintmax_t)index);
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
restart:
level = vm_radix_height(rtree, &rnode);
if (rnode == NULL)
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
return (NULL);
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
max = VM_RADIX_MAX(level);
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
if (index > max || index == 0)
index = max;
/*
* Search the tree from the top for any leaf node holding an index
* lower than 'index'.
*/
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level--;
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
while (rnode) {
slot = vm_radix_slot(index, level);
Fix a bug when the 'rnode' pointer can be NULL and we try to track the children. This helps in debugging case. Reported by: flo
2011-11-26 14:26:37 +00:00
CTR6(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"lookup_le: tree %p, index %ju, level %d, slot %d, rnode %p, child %p",
rtree, (uintmax_t)index, level, slot, rnode,
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
rnode->rn_child[slot]);
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
if (level == 0)
break;
/*
* If we don't have an exact match we must start our search
* from the next leaf and adjust our index appropriately.
*/
if ((child = rnode->rn_child[slot]) == NULL) {
/*
* Calculate how much to decrement our index by
* based on the tree level. We must set the
* lower bits to start from the end of the next
* leaf.
*/
inc = 1LL << (level * VM_RADIX_WIDTH);
index |= VM_RADIX_MAX(level);
index -= inc;
slot--;
CTR4(KTR_VM,
Fix format string for the pindex members as they should be treated as uintmax_t for compatibility among 32/64 bits.
2012-01-29 16:29:06 +00:00
"lookup_le: start %ju inc %ju mask 0x%jX slot %d",
(uintmax_t)index, (uintmax_t)inc,
(uintmax_t)VM_RADIX_MAX(level), slot);
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
for (; slot >= 0; slot--, index -= inc) {
child = rnode->rn_child[slot];
if (child)
break;
}
}
rnode = child;
level--;
}
if (rnode) {
for (; slot >= 0; slot--, index--) {
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
val = vm_radix_match(rnode->rn_child[slot], color);
if (val)
return (val);
- Implement vm_radix_lookup_le(). - Fix vm_radix_lookupn() when max == -1 by making the end parameter inclusive.
2011-10-23 01:19:01 +00:00
}
}
if (index != -1)
goto restart;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
return (NULL);
}
/*
* Remove the specified index from the tree. If possible the height of the
* tree is adjusted after deletion. The value stored at index is returned
* panics if the key is not present.
*/
void *
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
vm_radix_remove(struct vm_radix *rtree, vm_pindex_t index, int color)
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
{
struct vm_radix_node *stack[VM_RADIX_LIMIT];
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
struct vm_radix_node *rnode, *root;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
void *val;
int level;
int slot;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level = vm_radix_height(rtree, &root);
KASSERT(index <= VM_RADIX_MAX(level),
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
("vm_radix_remove: %p index %ju out of range %jd.",
rtree, (uintmax_t)index, VM_RADIX_MAX(level)));
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
rnode = root;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
val = NULL;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level--;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Find the node and record the path in stack.
*/
while (level && rnode) {
stack[level] = rnode;
slot = vm_radix_slot(index, level);
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR5(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"remove: tree %p, index %ju, level %d, slot %d, rnode %p",
rtree, (uintmax_t)index, level, slot, rnode);
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR4(KTR_VM, "remove: tree %p, rnode %p, child %p, count %u",
rtree, rnode, rnode->rn_child[slot], rnode->rn_count);
Improve the diagnostic in the remove case.
2011-11-28 17:26:19 +00:00
rnode = rnode->rn_child[slot];
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
level--;
}
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
KASSERT(rnode != NULL,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
("vm_radix_remove: index %ju not present in the tree.\n",
(uintmax_t)index));
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
slot = vm_radix_slot(index, 0);
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
val = vm_radix_match(rnode->rn_child[slot], color);
KASSERT(val != NULL,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
("vm_radix_remove: index %ju not present in the tree.\n",
(uintmax_t)index));
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
for (;;) {
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR5(KTR_VM,
Fix compilation issue on 32-bit targets. Reviewed by: attilio
2011-12-05 16:06:12 +00:00
"remove: resetting tree %p, index %ju, level %d, slot %d, rnode %p",
rtree, (uintmax_t)index, level, slot, rnode);
- Remove unnecessary checks on rnode in KTR prints - Track rn_count in KTR prints - Improve KTR in a way it best fits rn_count tracking
2011-11-29 02:07:07 +00:00
CTR4(KTR_VM,
"remove: resetting tree %p, rnode %p, child %p, count %u",
rtree, rnode,
(rnode != NULL) ? rnode->rn_child[slot] : NULL,
(rnode != NULL) ? rnode->rn_count : 0);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
rnode->rn_child[slot] = NULL;
Use atomics for rn_count on leaf node because RED operations happen without the VM_OBJECT_LOCK held, thus can be concurrent with BLACK ones. However, also use a write memory barrier in order to not reorder the operation of decrementing rn_count in respect fetching the pointer. Discussed with: jeff
2011-12-06 22:57:48 +00:00
/*
* Use atomics for the last level since red and black
* will both adjust it.
* Use a write memory barrier here in order to avoid
* rn_count reaching 0 before to fetch the actual pointer.
* Concurrent black removal, infact, may want to reclaim
* the radix node itself before to read it.
*/
if (level == 0)
atomic_add_rel_32(&rnode->rn_count, -1);
else
rnode->rn_count--;
- Support two types of nodes, red and black, within the same radix tree. Black nodes support standard active pages and red nodes support cached pages. Red nodes may be removed without the object lock but will not collapse unused tree nodes. Red nodes may not be directly inserted, instead a new function is supplied to convert between black and red. - Handle cached pages and active pages in the same loop in vm_object_split, vm_object_backing_scan, and vm_object_terminate. - Retire the splay page handling as the ifdefs are too difficult to maintain. - Slightly optimize the vm_radix_lookupn() function.
2011-10-30 11:11:04 +00:00
/*
* Only allow black removes to prune the tree.
*/
if ((color & VM_RADIX_BLACK) == 0 || rnode->rn_count > 0)
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
break;
vm_radix_node_put(rnode);
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
if (rnode == root) {
vm_radix_setroot(rtree, NULL, 0);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
break;
}
rnode = stack[++level];
slot = vm_radix_slot(index, level);
}
return (val);
}
vm_object_terminate() doesn't actually free the pages in the splay tree. Reclaim all the nodes related to the radix tree for a specified vm_object when calling vm_object_terminate() via the newly added interface vm_radix_reclaim_nodes(). The function is recursive, but we have a well-defined maximum depth, thus the amount of necessary stack can be easilly calculated. Reported by: alc Discussed and reviewed by: jeff
2011-11-01 03:40:38 +00:00
/*
* Remove and free all the nodes from the radix tree.
* This function is recrusive but there is a tight control on it as the
* maximum depth of the tree is fixed.
*/
void
vm_radix_reclaim_allnodes(struct vm_radix *rtree)
{
struct vm_radix_node *root;
int level;
if (rtree->rt_root == 0)
return;
level = vm_radix_height(rtree, &root);
vm_radix_reclaim_allnodes_internal(root, level - 1);
rtree->rt_root = 0;
}
- Exclude vm_radix_shrink() from the interface but retain the code still as it can be useful. - Make most of the interface private as it is unnecessary public right now. This will help in making nodes changing with arch and still avoid namespace pollution.
2012-03-01 00:54:08 +00:00
#ifdef notyet
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/*
* Attempts to reduce the height of the tree.
*/
void
vm_radix_shrink(struct vm_radix *rtree)
{
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
struct vm_radix_node *tmp, *root;
int level;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
if (rtree->rt_root == 0)
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
return;
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
level = vm_radix_height(rtree, &root);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
/* Adjust the height of the tree. */
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
while (root->rn_count == 1 && root->rn_child[0] != NULL) {
tmp = root;
root->rn_count--;
root = root->rn_child[0];
level--;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
vm_radix_node_put(tmp);
}
/* Finally see if we have an empty tree. */
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
if (root->rn_count == 0) {
vm_radix_node_put(root);
root = NULL;
level--;
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
- Use a single uintptr_t for the root of the radix node that encodes the height and a pointer so that the update to the root is atomic. This permits safe lookups in parallel with tree expansion. Shrinking the space requirements is a small bonus.
2011-10-28 03:42:41 +00:00
vm_radix_setroot(rtree, root, level);
Check in an intial implementation of radix tree implementation to replace the vm object pages splay. TODO: - Handle differently the negative keys for having smaller depth index nodes (negative keys caming from indirect blocks) - Fix the get_node() by having support for a low reserved objects directly from UMA - Implement the lookup_le and re-enable VM_NRESERVELEVEL = 1 - Try to rework the superpage splay of idle pages and the cache splay for every vm object in order to regain space on vm_page structure - Verify performance and improve them (likely by having consumers to deal with several ranges of pages manually?) Obtained from: jeff, Mayur Shardul (GSoC 2009)
2011-10-22 23:34:37 +00:00
}
- Exclude vm_radix_shrink() from the interface but retain the code still as it can be useful. - Make most of the interface private as it is unnecessary public right now. This will help in making nodes changing with arch and still avoid namespace pollution.
2012-03-01 00:54:08 +00:00
#endif
Reference in New Issue Copy Permalink