2005-01-07 02:29:27 +00:00
|
|
|
/*-
|
2017-11-30 15:48:35 +00:00
|
|
|
* SPDX-License-Identifier: (BSD-3-Clause AND MIT-CMU)
|
2017-11-20 19:43:44 +00:00
|
|
|
*
|
1994-05-25 09:21:21 +00:00
|
|
|
* Copyright (c) 1991 Regents of the University of California.
|
|
|
|
* All rights reserved.
|
2008-03-18 06:52:15 +00:00
|
|
|
* Copyright (c) 1998 Matthew Dillon. All Rights Reserved.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
|
|
|
* This code is derived from software contributed to Berkeley by
|
|
|
|
* The Mach Operating System project at Carnegie-Mellon University.
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2017-02-28 23:42:47 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
1994-05-24 10:09:53 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
* without specific prior written permission.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
*
|
1994-05-25 09:21:21 +00:00
|
|
|
* from: @(#)vm_page.c 7.4 (Berkeley) 5/7/91
|
|
|
|
*/
|
|
|
|
|
2005-01-07 02:29:27 +00:00
|
|
|
/*-
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1987, 1990 Carnegie-Mellon University.
|
|
|
|
* All rights reserved.
|
|
|
|
*
|
|
|
|
* Authors: Avadis Tevanian, Jr., Michael Wayne Young
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Permission to use, copy, modify and distribute this software and
|
|
|
|
* its documentation is hereby granted, provided that both the copyright
|
|
|
|
* notice and this permission notice appear in all copies of the
|
|
|
|
* software, derivative works or modified versions, and any portions
|
|
|
|
* thereof, and that both notices appear in supporting documentation.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
|
|
|
* CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS"
|
|
|
|
* CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND
|
1994-05-24 10:09:53 +00:00
|
|
|
* FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Carnegie Mellon requests users of this software to return to
|
|
|
|
*
|
|
|
|
* Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU
|
|
|
|
* School of Computer Science
|
|
|
|
* Carnegie Mellon University
|
|
|
|
* Pittsburgh PA 15213-3890
|
|
|
|
*
|
|
|
|
* any improvements or extensions that they make and grant Carnegie the
|
|
|
|
* rights to redistribute these changes.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Resident memory management module.
|
|
|
|
*/
|
|
|
|
|
2003-06-11 23:50:51 +00:00
|
|
|
#include <sys/cdefs.h>
|
2007-12-29 19:53:04 +00:00
|
|
|
#include "opt_vm.h"
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
|
|
|
#include <sys/systm.h>
|
2019-09-25 17:08:35 +00:00
|
|
|
#include <sys/counter.h>
|
2018-01-12 22:48:23 +00:00
|
|
|
#include <sys/domainset.h>
|
2005-08-12 12:24:19 +00:00
|
|
|
#include <sys/kernel.h>
|
2009-01-03 13:24:08 +00:00
|
|
|
#include <sys/limits.h>
|
2015-04-29 15:57:14 +00:00
|
|
|
#include <sys/linker.h>
|
2019-09-25 17:08:35 +00:00
|
|
|
#include <sys/lock.h>
|
1996-01-27 00:13:33 +00:00
|
|
|
#include <sys/malloc.h>
|
2013-06-10 01:48:21 +00:00
|
|
|
#include <sys/mman.h>
|
2010-05-16 19:25:56 +00:00
|
|
|
#include <sys/msgbuf.h>
|
2001-05-22 07:01:11 +00:00
|
|
|
#include <sys/mutex.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <sys/proc.h>
|
2013-03-09 02:32:23 +00:00
|
|
|
#include <sys/rwlock.h>
|
2019-09-10 18:27:45 +00:00
|
|
|
#include <sys/sleepqueue.h>
|
2015-04-29 15:57:14 +00:00
|
|
|
#include <sys/sbuf.h>
|
2018-04-24 21:15:54 +00:00
|
|
|
#include <sys/sched.h>
|
Autotune the number of pages set aside for UMA startup based on the number
of CPUs present. On amd64 this unbreaks the boot for systems with 92 or
more CPUs; the limit will vary on other systems depending on the size of
their uma_zone and uma_cache structures.
The major consumer of pages during UMA startup is the 19 zone structures
which are set up before UMA has bootstrapped itself sufficiently to use
the rest of the available memory: UMA Slabs, UMA Hash, 4 / 6 / 8 / 12 /
16 / 32 / 64 / 128 / 256 Bucket, vmem btag, VM OBJECT, RADIX NODE, MAP,
KMAP ENTRY, MAP ENTRY, VMSPACE, and fakepg. If the zone structures occupy
more than one page, they will not share pages and the number of pages
currently needed for startup is 19 * pages_per_zone + N, where N is the
number of pages used for allocating other structures; on amd64 N = 3 at
present (2 pages are allocated for UMA Kegs, and one page for UMA Hash).
This patch adds a new definition UMA_BOOT_PAGES_ZONES, currently set to 32,
and if a zone structure does not fit into a single page sets boot_pages to
UMA_BOOT_PAGES_ZONES * pages_per_zone instead of UMA_BOOT_PAGES (which
remains at 64). Consequently this patch has no effect on systems where the
zone structure fits into 2 or fewer pages (on amd64, 59 or fewer CPUs), but
increases boot_pages sufficiently on systems where the large number of CPUs
makes this structure larger. It seems safe to assume that systems with 60+
CPUs can afford to set aside an additional 128kB of memory per 32 CPUs.
The vm.boot_pages tunable continues to override this computation, but is
unlikely to be necessary in the future.
Tested on: EC2 x1.32xlarge
Relnotes: FreeBSD can now boot on 92+ CPU systems without requiring
vm.boot_pages to be manually adjusted.
Reviewed by: jeff, alc, adrian
Approved by: re (kib)
2016-07-07 18:37:12 +00:00
|
|
|
#include <sys/smp.h>
|
2005-08-12 12:24:19 +00:00
|
|
|
#include <sys/sysctl.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <sys/vmmeter.h>
|
1997-12-29 00:25:11 +00:00
|
|
|
#include <sys/vnode.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
#include <vm/vm.h>
|
2010-04-30 00:46:43 +00:00
|
|
|
#include <vm/pmap.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_param.h>
|
2018-01-12 22:48:23 +00:00
|
|
|
#include <vm/vm_domainset.h>
|
1995-03-16 18:17:34 +00:00
|
|
|
#include <vm/vm_kern.h>
|
2018-02-06 22:06:59 +00:00
|
|
|
#include <vm/vm_map.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_object.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_page.h>
|
|
|
|
#include <vm/vm_pageout.h>
|
Enable the new physical memory allocator.
This allocator uses a binary buddy system with a twist. First and
foremost, this allocator is required to support the implementation of
superpages. As a side effect, it enables a more robust implementation
of contigmalloc(9). Moreover, this reimplementation of
contigmalloc(9) eliminates the acquisition of Giant by
contigmalloc(..., M_NOWAIT, ...).
The twist is that this allocator tries to reduce the number of TLB
misses incurred by accesses through a direct map to small, UMA-managed
objects and page table pages. Roughly speaking, the physical pages
that are allocated for such purposes are clustered together in the
physical address space. The performance benefits vary. In the most
extreme case, a uniprocessor kernel running on an Opteron, I measured
an 18% reduction in system time during a buildworld.
This allocator does not implement page coloring. The reason is that
superpages have much the same effect. The contiguous physical memory
allocation necessary for a superpage is inherently colored.
Finally, the one caveat is that this allocator does not effectively
support prezeroed pages. I hope this is temporary. On i386, this is
a slight pessimization. However, on amd64, the beneficial effects of
the direct-map optimization outweigh the ill effects. I speculate
that this is true in general of machines with a direct map.
Approved by: re
2007-06-16 04:57:06 +00:00
|
|
|
#include <vm/vm_phys.h>
|
2018-02-06 22:10:07 +00:00
|
|
|
#include <vm/vm_pagequeue.h>
|
|
|
|
#include <vm/vm_pager.h>
|
Sync back vmcontention branch into HEAD:
Replace the per-object resident and cached pages splay tree with a
path-compressed multi-digit radix trie.
Along with this, switch also the x86-specific handling of idle page
tables to using the radix trie.
This change is supposed to do the following:
- Allowing the acquisition of read locking for lookup operations of the
resident/cached pages collections as the per-vm_page_t splay iterators
are now removed.
- Increase the scalability of the operations on the page collections.
The radix trie does rely on the consumers locking to ensure atomicity of
its operations. In order to avoid deadlocks the bisection nodes are
pre-allocated in the UMA zone. This can be done safely because the
algorithm needs at maximum one new node per insert which means the
maximum number of the desired nodes is the number of available physical
frames themselves. However, not all the times a new bisection node is
really needed.
The radix trie implements path-compression because UFS indirect blocks
can lead to several objects with a very sparse trie, increasing the number
of levels to usually scan. It also helps in the nodes pre-fetching by
introducing the single node per-insert property.
This code is not generalized (yet) because of the possible loss of
performance by having much of the sizes in play configurable.
However, efforts to make this code more general and then reusable in
further different consumers might be really done.
The only KPI change is the removal of the function vm_page_splay() which
is now reaped.
The only KBI change, instead, is the removal of the left/right iterators
from struct vm_page, which are now reaped.
Further technical notes broken into mealpieces can be retrieved from the
svn branch:
http://svn.freebsd.org/base/user/attilio/vmcontention/
Sponsored by: EMC / Isilon storage division
In collaboration with: alc, jeff
Tested by: flo, pho, jhb, davide
Tested by: ian (arm)
Tested by: andreast (powerpc)
2013-03-18 00:25:02 +00:00
|
|
|
#include <vm/vm_radix.h>
|
2007-12-29 19:53:04 +00:00
|
|
|
#include <vm/vm_reserv.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_extern.h>
|
2020-10-14 22:51:40 +00:00
|
|
|
#include <vm/vm_dumpset.h>
|
2002-03-19 09:11:49 +00:00
|
|
|
#include <vm/uma.h>
|
|
|
|
#include <vm/uma_int.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
Introduce minidumps. Full physical memory crash dumps are still available
via the debug.minidump sysctl and tunable.
Traditional dumps store all physical memory. This was once a good thing
when machines had a maximum of 64M of ram and 1GB of kvm. These days,
machines often have many gigabytes of ram and a smaller amount of kvm.
libkvm+kgdb don't have a way to access physical ram that is not mapped
into kvm at the time of the crash dump, so the extra ram being dumped
is mostly wasted.
Minidumps invert the process. Instead of dumping physical memory in
in order to guarantee that all of kvm's backing is dumped, minidumps
instead dump only memory that is actively mapped into kvm.
amd64 has a direct map region that things like UMA use. Obviously we
cannot dump all of the direct map region because that is effectively
an old style all-physical-memory dump. Instead, introduce a bitmap
and two helper routines (dump_add_page(pa) and dump_drop_page(pa)) that
allow certain critical direct map pages to be included in the dump.
uma_machdep.c's allocator is the intended consumer.
Dumps are a custom format. At the very beginning of the file is a header,
then a copy of the message buffer, then the bitmap of pages present in
the dump, then the final level of the kvm page table trees (2MB mappings
are expanded into a 4K page mappings), then the sparse physical pages
according to the bitmap. libkvm can now conveniently access the kvm
page table entries.
Booting my test 8GB machine, forcing it into ddb and forcing a dump
leads to a 48MB minidump. While this is a best case, I expect minidumps
to be in the 100MB-500MB range. Obviously, never larger than physical
memory of course.
minidumps are on by default. It would want be necessary to turn them off
if it was necessary to debug corrupt kernel page table management as that
would mess up minidumps as well.
Both minidumps and regular dumps are supported on the same machine.
2006-04-21 04:24:50 +00:00
|
|
|
#include <machine/md_var.h>
|
|
|
|
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
struct vm_domain vm_dom[MAXMEMDOM];
|
2002-07-04 22:07:37 +00:00
|
|
|
|
2018-07-05 17:13:37 +00:00
|
|
|
DPCPU_DEFINE_STATIC(struct vm_batchqueue, pqbatch[MAXMEMDOM][PQ_COUNT]);
|
2018-04-24 21:15:54 +00:00
|
|
|
|
2017-09-06 20:28:18 +00:00
|
|
|
struct mtx_padalign __exclusive_cache_line pa_lock[PA_LOCK_COUNT];
|
2018-03-15 19:23:07 +00:00
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
struct mtx_padalign __exclusive_cache_line vm_domainset_lock;
|
2018-03-15 19:23:07 +00:00
|
|
|
/* The following fields are protected by the domainset lock. */
|
2018-02-06 22:10:07 +00:00
|
|
|
domainset_t __exclusive_cache_line vm_min_domains;
|
|
|
|
domainset_t __exclusive_cache_line vm_severe_domains;
|
|
|
|
static int vm_min_waiters;
|
|
|
|
static int vm_severe_waiters;
|
|
|
|
static int vm_pageproc_waiters;
|
|
|
|
|
2020-02-26 14:26:36 +00:00
|
|
|
static SYSCTL_NODE(_vm_stats, OID_AUTO, page, CTLFLAG_RD | CTLFLAG_MPSAFE, 0,
|
2019-09-25 17:08:35 +00:00
|
|
|
"VM page statistics");
|
|
|
|
|
2020-03-06 19:10:00 +00:00
|
|
|
static COUNTER_U64_DEFINE_EARLY(pqstate_commit_retries);
|
2019-12-28 19:03:32 +00:00
|
|
|
SYSCTL_COUNTER_U64(_vm_stats_page, OID_AUTO, pqstate_commit_retries,
|
|
|
|
CTLFLAG_RD, &pqstate_commit_retries,
|
|
|
|
"Number of failed per-page atomic queue state updates");
|
|
|
|
|
2020-03-06 19:10:00 +00:00
|
|
|
static COUNTER_U64_DEFINE_EARLY(queue_ops);
|
2019-09-25 17:08:35 +00:00
|
|
|
SYSCTL_COUNTER_U64(_vm_stats_page, OID_AUTO, queue_ops,
|
|
|
|
CTLFLAG_RD, &queue_ops,
|
|
|
|
"Number of batched queue operations");
|
|
|
|
|
2020-03-06 19:10:00 +00:00
|
|
|
static COUNTER_U64_DEFINE_EARLY(queue_nops);
|
2019-09-25 17:08:35 +00:00
|
|
|
SYSCTL_COUNTER_U64(_vm_stats_page, OID_AUTO, queue_nops,
|
|
|
|
CTLFLAG_RD, &queue_nops,
|
|
|
|
"Number of batched queue operations with no effects");
|
|
|
|
|
2017-01-04 22:27:19 +00:00
|
|
|
/*
|
|
|
|
* bogus page -- for I/O to/from partially complete buffers,
|
|
|
|
* or for paging into sparsely invalid regions.
|
|
|
|
*/
|
|
|
|
vm_page_t bogus_page;
|
|
|
|
|
2012-05-12 20:03:06 +00:00
|
|
|
vm_page_t vm_page_array;
|
|
|
|
long vm_page_array_size;
|
|
|
|
long first_page;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2020-09-21 22:20:37 +00:00
|
|
|
struct bitset *vm_page_dump;
|
|
|
|
long vm_page_dump_pages;
|
|
|
|
|
2020-07-09 22:38:36 +00:00
|
|
|
static TAILQ_HEAD(, vm_page) blacklist_head;
|
|
|
|
static int sysctl_vm_page_blacklist(SYSCTL_HANDLER_ARGS);
|
|
|
|
SYSCTL_PROC(_vm, OID_AUTO, page_blacklist, CTLTYPE_STRING | CTLFLAG_RD |
|
|
|
|
CTLFLAG_MPSAFE, NULL, 0, sysctl_vm_page_blacklist, "A", "Blacklist pages");
|
2015-04-29 15:57:14 +00:00
|
|
|
|
2011-03-11 07:07:48 +00:00
|
|
|
static uma_zone_t fakepg_zone;
|
|
|
|
|
2016-11-15 18:22:50 +00:00
|
|
|
static void vm_page_alloc_check(vm_page_t m);
|
2019-12-24 18:38:06 +00:00
|
|
|
static bool _vm_page_busy_sleep(vm_object_t obj, vm_page_t m,
|
2020-02-17 01:08:00 +00:00
|
|
|
vm_pindex_t pindex, const char *wmesg, int allocflags, bool locked);
|
2011-11-05 08:20:32 +00:00
|
|
|
static void vm_page_clear_dirty_mask(vm_page_t m, vm_page_bits_t pagebits);
|
2018-04-24 21:15:54 +00:00
|
|
|
static void vm_page_enqueue(vm_page_t m, uint8_t queue);
|
2019-12-22 06:56:44 +00:00
|
|
|
static bool vm_page_free_prep(vm_page_t m);
|
|
|
|
static void vm_page_free_toq(vm_page_t m);
|
2017-01-04 22:27:19 +00:00
|
|
|
static void vm_page_init(void *dummy);
|
2013-08-09 11:28:55 +00:00
|
|
|
static int vm_page_insert_after(vm_page_t m, vm_object_t object,
|
2013-05-12 16:50:18 +00:00
|
|
|
vm_pindex_t pindex, vm_page_t mpred);
|
2013-08-09 11:28:55 +00:00
|
|
|
static void vm_page_insert_radixdone(vm_page_t m, vm_object_t object,
|
|
|
|
vm_page_t mpred);
|
2019-12-28 19:03:32 +00:00
|
|
|
static void vm_page_mvqueue(vm_page_t m, const uint8_t queue,
|
|
|
|
const uint16_t nflag);
|
2018-02-06 22:10:07 +00:00
|
|
|
static int vm_page_reclaim_run(int req_class, int domain, u_long npages,
|
|
|
|
vm_page_t m_run, vm_paddr_t high);
|
2019-12-28 19:03:32 +00:00
|
|
|
static void vm_page_release_toq(vm_page_t m, uint8_t nqueue, bool noreuse);
|
2018-02-06 22:10:07 +00:00
|
|
|
static int vm_domain_alloc_fail(struct vm_domain *vmd, vm_object_t object,
|
|
|
|
int req);
|
2019-07-20 18:25:41 +00:00
|
|
|
static int vm_page_zone_import(void *arg, void **store, int cnt, int domain,
|
2018-04-01 04:50:05 +00:00
|
|
|
int flags);
|
2019-07-20 18:25:41 +00:00
|
|
|
static void vm_page_zone_release(void *arg, void **store, int cnt);
|
2011-03-11 07:07:48 +00:00
|
|
|
|
2017-01-04 22:27:19 +00:00
|
|
|
SYSINIT(vm_page, SI_SUB_VM, SI_ORDER_SECOND, vm_page_init, NULL);
|
2011-03-11 07:07:48 +00:00
|
|
|
|
|
|
|
static void
|
2017-01-04 22:27:19 +00:00
|
|
|
vm_page_init(void *dummy)
|
2011-03-11 07:07:48 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
fakepg_zone = uma_zcreate("fakepg", sizeof(struct vm_page), NULL, NULL,
|
2020-02-19 08:17:27 +00:00
|
|
|
NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE);
|
2021-10-20 00:23:39 +00:00
|
|
|
bogus_page = vm_page_alloc_noobj(VM_ALLOC_WIRED);
|
2011-03-11 07:07:48 +00:00
|
|
|
}
|
2008-03-18 06:52:15 +00:00
|
|
|
|
2018-04-01 04:50:05 +00:00
|
|
|
/*
|
|
|
|
* The cache page zone is initialized later since we need to be able to allocate
|
|
|
|
* pages before UMA is fully initialized.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
vm_page_init_cache_zones(void *dummy __unused)
|
|
|
|
{
|
|
|
|
struct vm_domain *vmd;
|
2019-07-08 18:56:30 +00:00
|
|
|
struct vm_pgcache *pgcache;
|
2019-11-22 16:30:47 +00:00
|
|
|
int cache, domain, maxcache, pool;
|
2019-07-08 18:56:30 +00:00
|
|
|
|
2019-10-24 18:39:05 +00:00
|
|
|
maxcache = 0;
|
2020-01-10 19:32:08 +00:00
|
|
|
TUNABLE_INT_FETCH("vm.pgcache_zone_max_pcpu", &maxcache);
|
|
|
|
maxcache *= mp_ncpus;
|
2019-07-08 18:56:30 +00:00
|
|
|
for (domain = 0; domain < vm_ndomains; domain++) {
|
|
|
|
vmd = VM_DOMAIN(domain);
|
|
|
|
for (pool = 0; pool < VM_NFREEPOOL; pool++) {
|
|
|
|
pgcache = &vmd->vmd_pgcache[pool];
|
|
|
|
pgcache->domain = domain;
|
|
|
|
pgcache->pool = pool;
|
|
|
|
pgcache->zone = uma_zcache_create("vm pgcache",
|
2019-11-22 16:30:47 +00:00
|
|
|
PAGE_SIZE, NULL, NULL, NULL, NULL,
|
2019-07-20 18:25:41 +00:00
|
|
|
vm_page_zone_import, vm_page_zone_release, pgcache,
|
2019-11-22 16:30:47 +00:00
|
|
|
UMA_ZONE_VM);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Limit each pool's zone to 0.1% of the pages in the
|
|
|
|
* domain.
|
|
|
|
*/
|
|
|
|
cache = maxcache != 0 ? maxcache :
|
|
|
|
vmd->vmd_page_count / 1000;
|
|
|
|
uma_zone_set_maxcache(pgcache->zone, cache);
|
2019-07-08 18:56:30 +00:00
|
|
|
}
|
2018-04-01 04:50:05 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
SYSINIT(vm_page2, SI_SUB_VM_CONF, SI_ORDER_ANY, vm_page_init_cache_zones, NULL);
|
|
|
|
|
2008-09-26 18:44:40 +00:00
|
|
|
/* Make sure that u_long is at least 64 bits when PAGE_SIZE is 32K. */
|
|
|
|
#if PAGE_SIZE == 32768
|
|
|
|
#ifdef CTASSERT
|
|
|
|
CTASSERT(sizeof(u_long) >= 8);
|
|
|
|
#endif
|
|
|
|
#endif
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_set_page_size:
|
|
|
|
*
|
|
|
|
* Sets the page size, perhaps based upon the memory
|
|
|
|
* size. Must be called before any use of page-size
|
|
|
|
* dependent functions.
|
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_set_page_size(void)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2014-03-22 10:26:09 +00:00
|
|
|
if (vm_cnt.v_page_size == 0)
|
|
|
|
vm_cnt.v_page_size = PAGE_SIZE;
|
|
|
|
if (((vm_cnt.v_page_size - 1) & vm_cnt.v_page_size) != 0)
|
1994-05-24 10:09:53 +00:00
|
|
|
panic("vm_set_page_size: page size not a power of two");
|
|
|
|
}
|
|
|
|
|
2006-06-23 16:44:24 +00:00
|
|
|
/*
|
2020-07-09 22:38:36 +00:00
|
|
|
* vm_page_blacklist_next:
|
2006-06-23 16:44:24 +00:00
|
|
|
*
|
2020-07-09 22:38:36 +00:00
|
|
|
* Find the next entry in the provided string of blacklist
|
2015-04-29 15:57:14 +00:00
|
|
|
* addresses. Entries are separated by space, comma, or newline.
|
|
|
|
* If an invalid integer is encountered then the rest of the
|
|
|
|
* string is skipped. Updates the list pointer to the next
|
|
|
|
* character, or NULL if the string is exhausted or invalid.
|
2006-06-23 16:44:24 +00:00
|
|
|
*/
|
2015-04-29 15:57:14 +00:00
|
|
|
static vm_paddr_t
|
2020-07-09 22:38:36 +00:00
|
|
|
vm_page_blacklist_next(char **list, char *end)
|
2006-06-23 16:44:24 +00:00
|
|
|
{
|
|
|
|
vm_paddr_t bad;
|
|
|
|
char *cp, *pos;
|
|
|
|
|
2015-04-29 15:57:14 +00:00
|
|
|
if (list == NULL || *list == NULL)
|
|
|
|
return (0);
|
|
|
|
if (**list =='\0') {
|
|
|
|
*list = NULL;
|
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If there's no end pointer then the buffer is coming from
|
|
|
|
* the kenv and we know it's null-terminated.
|
|
|
|
*/
|
|
|
|
if (end == NULL)
|
|
|
|
end = *list + strlen(*list);
|
|
|
|
|
|
|
|
/* Ensure that strtoq() won't walk off the end */
|
|
|
|
if (*end != '\0') {
|
|
|
|
if (*end == '\n' || *end == ' ' || *end == ',')
|
|
|
|
*end = '\0';
|
|
|
|
else {
|
|
|
|
printf("Blacklist not terminated, skipping\n");
|
|
|
|
*list = NULL;
|
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
for (pos = *list; *pos != '\0'; pos = cp) {
|
2006-06-23 16:44:24 +00:00
|
|
|
bad = strtoq(pos, &cp, 0);
|
2015-04-29 15:57:14 +00:00
|
|
|
if (*cp == '\0' || *cp == ' ' || *cp == ',' || *cp == '\n') {
|
|
|
|
if (bad == 0) {
|
|
|
|
if (++cp < end)
|
2006-06-23 16:44:24 +00:00
|
|
|
continue;
|
2015-04-29 15:57:14 +00:00
|
|
|
else
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
break;
|
|
|
|
if (*cp == '\0' || ++cp >= end)
|
|
|
|
*list = NULL;
|
|
|
|
else
|
|
|
|
*list = cp;
|
|
|
|
return (trunc_page(bad));
|
2006-06-23 16:44:24 +00:00
|
|
|
}
|
2020-07-09 22:38:36 +00:00
|
|
|
printf("Garbage in RAM blacklist, skipping\n");
|
2015-04-29 15:57:14 +00:00
|
|
|
*list = NULL;
|
2006-06-23 16:44:24 +00:00
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
|
2018-04-07 17:06:13 +00:00
|
|
|
bool
|
2020-07-09 22:38:36 +00:00
|
|
|
vm_page_blacklist_add(vm_paddr_t pa, bool verbose)
|
2018-04-07 17:06:13 +00:00
|
|
|
{
|
|
|
|
struct vm_domain *vmd;
|
|
|
|
vm_page_t m;
|
2023-06-05 14:40:15 +00:00
|
|
|
bool found;
|
2018-04-07 17:06:13 +00:00
|
|
|
|
|
|
|
m = vm_phys_paddr_to_vm_page(pa);
|
|
|
|
if (m == NULL)
|
|
|
|
return (true); /* page does not exist, no failure */
|
|
|
|
|
|
|
|
vmd = vm_pagequeue_domain(m);
|
|
|
|
vm_domain_free_lock(vmd);
|
2023-06-05 14:40:15 +00:00
|
|
|
found = vm_phys_unfree_page(m);
|
2018-04-07 17:06:13 +00:00
|
|
|
vm_domain_free_unlock(vmd);
|
2023-06-05 14:40:15 +00:00
|
|
|
if (found) {
|
2018-11-29 16:31:01 +00:00
|
|
|
vm_domain_freecnt_inc(vmd, -1);
|
2020-07-09 22:38:36 +00:00
|
|
|
TAILQ_INSERT_TAIL(&blacklist_head, m, listq);
|
2018-04-07 17:06:13 +00:00
|
|
|
if (verbose)
|
|
|
|
printf("Skipping page with pa 0x%jx\n", (uintmax_t)pa);
|
|
|
|
}
|
2023-06-05 14:40:15 +00:00
|
|
|
return (found);
|
2018-04-07 17:06:13 +00:00
|
|
|
}
|
|
|
|
|
2015-04-29 15:57:14 +00:00
|
|
|
/*
|
2020-07-09 22:38:36 +00:00
|
|
|
* vm_page_blacklist_check:
|
2015-04-29 15:57:14 +00:00
|
|
|
*
|
2020-07-09 22:38:36 +00:00
|
|
|
* Iterate through the provided string of blacklist addresses, pulling
|
2015-04-29 15:57:14 +00:00
|
|
|
* each entry out of the physical allocator free list and putting it
|
2020-07-09 22:38:36 +00:00
|
|
|
* onto a list for reporting via the vm.page_blacklist sysctl.
|
2015-04-29 15:57:14 +00:00
|
|
|
*/
|
|
|
|
static void
|
2020-07-09 22:38:36 +00:00
|
|
|
vm_page_blacklist_check(char *list, char *end)
|
2015-04-29 15:57:14 +00:00
|
|
|
{
|
|
|
|
vm_paddr_t pa;
|
|
|
|
char *next;
|
|
|
|
|
|
|
|
next = list;
|
|
|
|
while (next != NULL) {
|
2020-07-09 22:38:36 +00:00
|
|
|
if ((pa = vm_page_blacklist_next(&next, end)) == 0)
|
2015-04-29 15:57:14 +00:00
|
|
|
continue;
|
2020-07-09 22:38:36 +00:00
|
|
|
vm_page_blacklist_add(pa, bootverbose);
|
2015-04-29 15:57:14 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2020-07-09 22:38:36 +00:00
|
|
|
* vm_page_blacklist_load:
|
2015-04-29 15:57:14 +00:00
|
|
|
*
|
2020-07-09 22:38:36 +00:00
|
|
|
* Search for a special module named "ram_blacklist". It'll be a
|
2015-04-29 15:57:14 +00:00
|
|
|
* plain text file provided by the user via the loader directive
|
|
|
|
* of the same name.
|
|
|
|
*/
|
|
|
|
static void
|
2020-07-09 22:38:36 +00:00
|
|
|
vm_page_blacklist_load(char **list, char **end)
|
2015-04-29 15:57:14 +00:00
|
|
|
{
|
|
|
|
void *mod;
|
|
|
|
u_char *ptr;
|
|
|
|
u_int len;
|
|
|
|
|
|
|
|
mod = NULL;
|
|
|
|
ptr = NULL;
|
|
|
|
|
2020-07-09 22:38:36 +00:00
|
|
|
mod = preload_search_by_type("ram_blacklist");
|
2015-04-29 15:57:14 +00:00
|
|
|
if (mod != NULL) {
|
|
|
|
ptr = preload_fetch_addr(mod);
|
|
|
|
len = preload_fetch_size(mod);
|
|
|
|
}
|
|
|
|
*list = ptr;
|
|
|
|
if (ptr != NULL)
|
|
|
|
*end = ptr + len;
|
|
|
|
else
|
|
|
|
*end = NULL;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int
|
2020-07-09 22:38:36 +00:00
|
|
|
sysctl_vm_page_blacklist(SYSCTL_HANDLER_ARGS)
|
2015-04-29 15:57:14 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
struct sbuf sbuf;
|
|
|
|
int error, first;
|
|
|
|
|
|
|
|
first = 1;
|
|
|
|
error = sysctl_wire_old_buffer(req, 0);
|
|
|
|
if (error != 0)
|
|
|
|
return (error);
|
|
|
|
sbuf_new_for_sysctl(&sbuf, NULL, 128, req);
|
2020-07-09 22:38:36 +00:00
|
|
|
TAILQ_FOREACH(m, &blacklist_head, listq) {
|
2015-04-29 15:57:14 +00:00
|
|
|
sbuf_printf(&sbuf, "%s%#jx", first ? "" : ",",
|
|
|
|
(uintmax_t)m->phys_addr);
|
|
|
|
first = 0;
|
|
|
|
}
|
|
|
|
error = sbuf_finish(&sbuf);
|
|
|
|
sbuf_delete(&sbuf);
|
|
|
|
return (error);
|
|
|
|
}
|
|
|
|
|
2018-04-19 14:09:44 +00:00
|
|
|
/*
|
|
|
|
* Initialize a dummy page for use in scans of the specified paging queue.
|
|
|
|
* In principle, this function only needs to set the flag PG_MARKER.
|
2019-07-08 19:46:20 +00:00
|
|
|
* Nonetheless, it write busies the page as a safety precaution.
|
2018-04-19 14:09:44 +00:00
|
|
|
*/
|
2020-08-11 20:37:45 +00:00
|
|
|
void
|
2019-11-20 18:00:43 +00:00
|
|
|
vm_page_init_marker(vm_page_t marker, int queue, uint16_t aflags)
|
2018-04-19 14:09:44 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
bzero(marker, sizeof(*marker));
|
2019-09-16 15:03:12 +00:00
|
|
|
marker->flags = PG_MARKER;
|
2019-12-10 18:14:50 +00:00
|
|
|
marker->a.flags = aflags;
|
2019-11-24 19:12:23 +00:00
|
|
|
marker->busy_lock = VPB_CURTHREAD_EXCLUSIVE;
|
2019-12-10 18:14:50 +00:00
|
|
|
marker->a.queue = queue;
|
2018-04-19 14:09:44 +00:00
|
|
|
}
|
|
|
|
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
static void
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_page_domain_init(int domain)
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
{
|
2018-02-06 22:10:07 +00:00
|
|
|
struct vm_domain *vmd;
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
struct vm_pagequeue *pq;
|
|
|
|
int i;
|
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
vmd = VM_DOMAIN(domain);
|
|
|
|
bzero(vmd, sizeof(*vmd));
|
2020-02-23 03:32:04 +00:00
|
|
|
*__DECONST(const char **, &vmd->vmd_pagequeues[PQ_INACTIVE].pq_name) =
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
"vm inactive pagequeue";
|
2020-02-23 03:32:04 +00:00
|
|
|
*__DECONST(const char **, &vmd->vmd_pagequeues[PQ_ACTIVE].pq_name) =
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
"vm active pagequeue";
|
2020-02-23 03:32:04 +00:00
|
|
|
*__DECONST(const char **, &vmd->vmd_pagequeues[PQ_LAUNDRY].pq_name) =
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
"vm laundry pagequeue";
|
2020-02-23 03:32:04 +00:00
|
|
|
*__DECONST(const char **,
|
|
|
|
&vmd->vmd_pagequeues[PQ_UNSWAPPABLE].pq_name) =
|
2017-01-03 00:05:44 +00:00
|
|
|
"vm unswappable pagequeue";
|
2018-02-06 22:10:07 +00:00
|
|
|
vmd->vmd_domain = domain;
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
vmd->vmd_page_count = 0;
|
|
|
|
vmd->vmd_free_count = 0;
|
|
|
|
vmd->vmd_segs = 0;
|
|
|
|
vmd->vmd_oom = FALSE;
|
|
|
|
for (i = 0; i < PQ_COUNT; i++) {
|
|
|
|
pq = &vmd->vmd_pagequeues[i];
|
|
|
|
TAILQ_INIT(&pq->pq_pl);
|
|
|
|
mtx_init(&pq->pq_mutex, pq->pq_name, "vm pagequeue",
|
|
|
|
MTX_DEF | MTX_DUPOK);
|
2018-08-23 21:03:45 +00:00
|
|
|
pq->pq_pdpages = 0;
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_page_init_marker(&vmd->vmd_markers[i], i, 0);
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
}
|
2018-02-06 22:10:07 +00:00
|
|
|
mtx_init(&vmd->vmd_free_mtx, "vm page free queue", NULL, MTX_DEF);
|
2018-03-15 19:23:07 +00:00
|
|
|
mtx_init(&vmd->vmd_pageout_mtx, "vm pageout lock", NULL, MTX_DEF);
|
2018-04-24 21:15:54 +00:00
|
|
|
snprintf(vmd->vmd_name, sizeof(vmd->vmd_name), "%d", domain);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* inacthead is used to provide FIFO ordering for LRU-bypassing
|
|
|
|
* insertions.
|
|
|
|
*/
|
|
|
|
vm_page_init_marker(&vmd->vmd_inacthead, PQ_INACTIVE, PGA_ENQUEUED);
|
2018-04-19 14:09:44 +00:00
|
|
|
TAILQ_INSERT_HEAD(&vmd->vmd_pagequeues[PQ_INACTIVE].pq_pl,
|
|
|
|
&vmd->vmd_inacthead, plinks.q);
|
2018-04-24 21:15:54 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The clock pages are used to implement active queue scanning without
|
|
|
|
* requeues. Scans start at clock[0], which is advanced after the scan
|
|
|
|
* ends. When the two clock hands meet, they are reset and scanning
|
|
|
|
* resumes from the head of the queue.
|
|
|
|
*/
|
|
|
|
vm_page_init_marker(&vmd->vmd_clock[0], PQ_ACTIVE, PGA_ENQUEUED);
|
|
|
|
vm_page_init_marker(&vmd->vmd_clock[1], PQ_ACTIVE, PGA_ENQUEUED);
|
|
|
|
TAILQ_INSERT_HEAD(&vmd->vmd_pagequeues[PQ_ACTIVE].pq_pl,
|
|
|
|
&vmd->vmd_clock[0], plinks.q);
|
|
|
|
TAILQ_INSERT_TAIL(&vmd->vmd_pagequeues[PQ_ACTIVE].pq_pl,
|
|
|
|
&vmd->vmd_clock[1], plinks.q);
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
}
|
|
|
|
|
2017-11-26 19:17:55 +00:00
|
|
|
/*
|
|
|
|
* Initialize a physical page in preparation for adding it to the free
|
|
|
|
* lists.
|
|
|
|
*/
|
2021-06-17 10:58:13 +00:00
|
|
|
void
|
2017-11-27 17:46:38 +00:00
|
|
|
vm_page_init_page(vm_page_t m, vm_paddr_t pa, int segind)
|
2017-11-26 19:17:55 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
m->object = NULL;
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
m->ref_count = 0;
|
2020-02-04 20:33:01 +00:00
|
|
|
m->busy_lock = VPB_FREED;
|
2019-12-10 18:14:50 +00:00
|
|
|
m->flags = m->a.flags = 0;
|
2017-11-26 19:17:55 +00:00
|
|
|
m->phys_addr = pa;
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.queue = PQ_NONE;
|
2017-11-26 19:17:55 +00:00
|
|
|
m->psind = 0;
|
2017-11-27 17:46:38 +00:00
|
|
|
m->segind = segind;
|
2017-11-26 19:17:55 +00:00
|
|
|
m->order = VM_NFREEORDER;
|
|
|
|
m->pool = VM_FREEPOOL_DEFAULT;
|
|
|
|
m->valid = m->dirty = 0;
|
|
|
|
pmap_page_init(m);
|
|
|
|
}
|
|
|
|
|
2019-08-18 23:07:56 +00:00
|
|
|
#ifndef PMAP_HAS_PAGE_ARRAY
|
|
|
|
static vm_paddr_t
|
|
|
|
vm_page_array_alloc(vm_offset_t *vaddr, vm_paddr_t end, vm_paddr_t page_range)
|
|
|
|
{
|
|
|
|
vm_paddr_t new_end;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reserve an unmapped guard page to trap access to vm_page_array[-1].
|
|
|
|
* However, because this page is allocated from KVM, out-of-bounds
|
|
|
|
* accesses using the direct map will not be trapped.
|
|
|
|
*/
|
|
|
|
*vaddr += PAGE_SIZE;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Allocate physical memory for the page structures, and map it.
|
|
|
|
*/
|
|
|
|
new_end = trunc_page(end - page_range * sizeof(struct vm_page));
|
|
|
|
vm_page_array = (vm_page_t)pmap_map(vaddr, new_end, end,
|
|
|
|
VM_PROT_READ | VM_PROT_WRITE);
|
|
|
|
vm_page_array_size = page_range;
|
|
|
|
|
|
|
|
return (new_end);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_page_startup:
|
|
|
|
*
|
2017-02-04 05:23:10 +00:00
|
|
|
* Initializes the resident memory module. Allocates physical memory for
|
|
|
|
* bootstrapping UMA and some data structures that are used to manage
|
|
|
|
* physical pages. Initializes these structures, and populates the free
|
|
|
|
* page queues.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1994-05-25 09:21:21 +00:00
|
|
|
vm_offset_t
|
2004-04-04 23:33:36 +00:00
|
|
|
vm_page_startup(vm_offset_t vaddr)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2017-09-07 21:43:39 +00:00
|
|
|
struct vm_phys_seg *seg;
|
2021-09-14 12:25:37 +00:00
|
|
|
struct vm_domain *vmd;
|
2017-09-07 21:43:39 +00:00
|
|
|
vm_page_t m;
|
2015-04-29 15:57:14 +00:00
|
|
|
char *list, *listend;
|
powerpc/pmap: NUMA-ize vm_page_array on powerpc
Summary:
This matches r351198 from amd64. This only applies to AIM64 and Book-E.
On AIM64 it short-circuits with one domain, to behave similar to
existing. Otherwise it will allocate 16MB huge pages to hold the page
array, across all NUMA domains. On the first domain it will shift the
page array base up, to "upper-align" the page array in that domain, so
as to reduce the number of pages from the next domain appearing in this
domain. After the first domain, subsequent domains will be allocated in
full 16MB pages, until the final domain, which can be short. This means
some inner domains may have pages accounted in earlier domains.
On Book-E the page array is setup at MMU bootstrap time so that it's
always mapped in TLB1, on both 32-bit and 64-bit. This reduces the TLB0
overhead for touching the vm_page_array, which reduces up to one TLB
miss per array access.
Since page_range (vm_page_startup()) is no longer used on Book-E but is on
32-bit AIM, mark the variable as potentially unused, rather than using a
nasty #if defined() list.
Reviewed by: luporl
Differential Revision: https://reviews.freebsd.org/D21449
2019-12-07 03:34:03 +00:00
|
|
|
vm_paddr_t end, high_avail, low_avail, new_end, size;
|
|
|
|
vm_paddr_t page_range __unused;
|
2021-09-14 12:25:37 +00:00
|
|
|
vm_paddr_t last_pa, pa, startp, endp;
|
2017-09-07 21:43:39 +00:00
|
|
|
u_long pagecount;
|
2020-09-21 22:20:37 +00:00
|
|
|
#if MINIDUMP_PAGE_TRACKING
|
|
|
|
u_long vm_page_dump_size;
|
|
|
|
#endif
|
Followup on r302393 by cperciva, improving calculation of boot pages required
for UMA startup.
o Introduce another stage of UMA startup, which is entered after
vm_page_startup() finishes. After this stage we don't yet enable buckets,
but we can ask VM for pages. Rename stages to meaningful names while here.
New list of stages: BOOT_COLD, BOOT_STRAPPED, BOOT_PAGEALLOC, BOOT_BUCKETS,
BOOT_RUNNING.
Enabling page alloc earlier allows us to dramatically reduce number of
boot pages required. What is more important number of zones becomes
consistent across different machines, as no MD allocations are done before
the BOOT_PAGEALLOC stage. Now only UMA internal zones actually need to use
startup_alloc(), however that may change, so vm_page_startup() provides
its need for early zones as argument.
o Introduce uma_startup_count() function, to avoid code duplication. The
functions calculates sizes of zones zone and kegs zone, and calculates how
many pages UMA will need to bootstrap.
It counts not only of zone structures, but also of kegs, slabs and hashes.
o Hide uma_startup_foo() declarations from public file.
o Provide several DIAGNOSTIC printfs on boot_pages usage.
o Bugfix: when calculating zone of zones size use (mp_maxid + 1) instead of
mp_ncpus. Use resulting number not only in the size argument to zone_ctor()
but also as args.size.
Reviewed by: imp, gallatin (earlier version)
Differential Revision: https://reviews.freebsd.org/D14054
2018-02-06 04:16:00 +00:00
|
|
|
int biggestone, i, segind;
|
2019-01-07 23:17:09 +00:00
|
|
|
#ifdef WITNESS
|
2020-01-16 05:01:21 +00:00
|
|
|
vm_offset_t mapped;
|
2019-01-07 23:17:09 +00:00
|
|
|
int witness_size;
|
|
|
|
#endif
|
2018-07-05 16:43:15 +00:00
|
|
|
#if defined(__i386__) && defined(VM_PHYSSEG_DENSE)
|
|
|
|
long ii;
|
|
|
|
#endif
|
1994-05-25 09:21:21 +00:00
|
|
|
|
|
|
|
vaddr = round_page(vaddr);
|
|
|
|
|
2019-08-18 07:06:31 +00:00
|
|
|
vm_phys_early_startup();
|
|
|
|
biggestone = vm_phys_avail_largest();
|
2001-03-01 19:21:24 +00:00
|
|
|
end = phys_avail[biggestone+1];
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-07-04 22:07:37 +00:00
|
|
|
/*
|
2012-06-27 03:45:25 +00:00
|
|
|
* Initialize the page and queue locks.
|
2002-07-04 22:07:37 +00:00
|
|
|
*/
|
2018-02-06 22:10:07 +00:00
|
|
|
mtx_init(&vm_domainset_lock, "vm domainset lock", NULL, MTX_DEF);
|
2010-04-30 00:46:43 +00:00
|
|
|
for (i = 0; i < PA_LOCK_COUNT; i++)
|
2012-10-31 18:07:18 +00:00
|
|
|
mtx_init(&pa_lock[i], "vm page", NULL, MTX_DEF);
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
for (i = 0; i < vm_ndomains; i++)
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_page_domain_init(i);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2020-01-16 05:01:21 +00:00
|
|
|
new_end = end;
|
2018-03-22 19:11:43 +00:00
|
|
|
#ifdef WITNESS
|
2019-01-07 23:17:09 +00:00
|
|
|
witness_size = round_page(witness_startup_count());
|
|
|
|
new_end -= witness_size;
|
|
|
|
mapped = pmap_map(&vaddr, new_end, new_end + witness_size,
|
2018-03-22 19:11:43 +00:00
|
|
|
VM_PROT_READ | VM_PROT_WRITE);
|
2019-01-07 23:17:09 +00:00
|
|
|
bzero((void *)mapped, witness_size);
|
2018-03-22 19:11:43 +00:00
|
|
|
witness_startup((void *)mapped);
|
|
|
|
#endif
|
|
|
|
|
2020-09-21 22:20:37 +00:00
|
|
|
#if MINIDUMP_PAGE_TRACKING
|
Introduce minidumps. Full physical memory crash dumps are still available
via the debug.minidump sysctl and tunable.
Traditional dumps store all physical memory. This was once a good thing
when machines had a maximum of 64M of ram and 1GB of kvm. These days,
machines often have many gigabytes of ram and a smaller amount of kvm.
libkvm+kgdb don't have a way to access physical ram that is not mapped
into kvm at the time of the crash dump, so the extra ram being dumped
is mostly wasted.
Minidumps invert the process. Instead of dumping physical memory in
in order to guarantee that all of kvm's backing is dumped, minidumps
instead dump only memory that is actively mapped into kvm.
amd64 has a direct map region that things like UMA use. Obviously we
cannot dump all of the direct map region because that is effectively
an old style all-physical-memory dump. Instead, introduce a bitmap
and two helper routines (dump_add_page(pa) and dump_drop_page(pa)) that
allow certain critical direct map pages to be included in the dump.
uma_machdep.c's allocator is the intended consumer.
Dumps are a custom format. At the very beginning of the file is a header,
then a copy of the message buffer, then the bitmap of pages present in
the dump, then the final level of the kvm page table trees (2MB mappings
are expanded into a 4K page mappings), then the sparse physical pages
according to the bitmap. libkvm can now conveniently access the kvm
page table entries.
Booting my test 8GB machine, forcing it into ddb and forcing a dump
leads to a 48MB minidump. While this is a best case, I expect minidumps
to be in the 100MB-500MB range. Obviously, never larger than physical
memory of course.
minidumps are on by default. It would want be necessary to turn them off
if it was necessary to debug corrupt kernel page table management as that
would mess up minidumps as well.
Both minidumps and regular dumps are supported on the same machine.
2006-04-21 04:24:50 +00:00
|
|
|
/*
|
|
|
|
* Allocate a bitmap to indicate that a random physical page
|
|
|
|
* needs to be included in a minidump.
|
|
|
|
*
|
|
|
|
* The amd64 port needs this to indicate which direct map pages
|
|
|
|
* need to be dumped, via calls to dump_add_page()/dump_drop_page().
|
|
|
|
*
|
|
|
|
* However, i386 still needs this workspace internally within the
|
|
|
|
* minidump code. In theory, they are not needed on i386, but are
|
|
|
|
* included should the sf_buf code decide to use them.
|
|
|
|
*/
|
2010-12-01 03:35:19 +00:00
|
|
|
last_pa = 0;
|
2020-09-21 22:21:59 +00:00
|
|
|
vm_page_dump_pages = 0;
|
|
|
|
for (i = 0; dump_avail[i + 1] != 0; i += 2) {
|
|
|
|
vm_page_dump_pages += howmany(dump_avail[i + 1], PAGE_SIZE) -
|
|
|
|
dump_avail[i] / PAGE_SIZE;
|
2010-12-01 03:35:19 +00:00
|
|
|
if (dump_avail[i + 1] > last_pa)
|
|
|
|
last_pa = dump_avail[i + 1];
|
2020-09-21 22:21:59 +00:00
|
|
|
}
|
2020-09-21 22:20:37 +00:00
|
|
|
vm_page_dump_size = round_page(BITSET_SIZE(vm_page_dump_pages));
|
Introduce minidumps. Full physical memory crash dumps are still available
via the debug.minidump sysctl and tunable.
Traditional dumps store all physical memory. This was once a good thing
when machines had a maximum of 64M of ram and 1GB of kvm. These days,
machines often have many gigabytes of ram and a smaller amount of kvm.
libkvm+kgdb don't have a way to access physical ram that is not mapped
into kvm at the time of the crash dump, so the extra ram being dumped
is mostly wasted.
Minidumps invert the process. Instead of dumping physical memory in
in order to guarantee that all of kvm's backing is dumped, minidumps
instead dump only memory that is actively mapped into kvm.
amd64 has a direct map region that things like UMA use. Obviously we
cannot dump all of the direct map region because that is effectively
an old style all-physical-memory dump. Instead, introduce a bitmap
and two helper routines (dump_add_page(pa) and dump_drop_page(pa)) that
allow certain critical direct map pages to be included in the dump.
uma_machdep.c's allocator is the intended consumer.
Dumps are a custom format. At the very beginning of the file is a header,
then a copy of the message buffer, then the bitmap of pages present in
the dump, then the final level of the kvm page table trees (2MB mappings
are expanded into a 4K page mappings), then the sparse physical pages
according to the bitmap. libkvm can now conveniently access the kvm
page table entries.
Booting my test 8GB machine, forcing it into ddb and forcing a dump
leads to a 48MB minidump. While this is a best case, I expect minidumps
to be in the 100MB-500MB range. Obviously, never larger than physical
memory of course.
minidumps are on by default. It would want be necessary to turn them off
if it was necessary to debug corrupt kernel page table management as that
would mess up minidumps as well.
Both minidumps and regular dumps are supported on the same machine.
2006-04-21 04:24:50 +00:00
|
|
|
new_end -= vm_page_dump_size;
|
|
|
|
vm_page_dump = (void *)(uintptr_t)pmap_map(&vaddr, new_end,
|
|
|
|
new_end + vm_page_dump_size, VM_PROT_READ | VM_PROT_WRITE);
|
|
|
|
bzero((void *)vm_page_dump, vm_page_dump_size);
|
2017-09-07 21:43:39 +00:00
|
|
|
#else
|
|
|
|
(void)last_pa;
|
2010-05-16 19:25:56 +00:00
|
|
|
#endif
|
2022-01-06 00:42:40 +00:00
|
|
|
#if defined(__aarch64__) || defined(__amd64__) || \
|
2019-10-14 13:04:04 +00:00
|
|
|
defined(__riscv) || defined(__powerpc64__)
|
2017-02-04 05:23:10 +00:00
|
|
|
/*
|
2019-01-07 23:17:09 +00:00
|
|
|
* Include the UMA bootstrap pages, witness pages and vm_page_dump
|
|
|
|
* in a crash dump. When pmap_map() uses the direct map, they are
|
|
|
|
* not automatically included.
|
2017-02-04 05:23:10 +00:00
|
|
|
*/
|
|
|
|
for (pa = new_end; pa < end; pa += PAGE_SIZE)
|
|
|
|
dump_add_page(pa);
|
|
|
|
#endif
|
|
|
|
phys_avail[biggestone + 1] = new_end;
|
2010-05-16 19:25:56 +00:00
|
|
|
#ifdef __amd64__
|
|
|
|
/*
|
|
|
|
* Request that the physical pages underlying the message buffer be
|
|
|
|
* included in a crash dump. Since the message buffer is accessed
|
|
|
|
* through the direct map, they are not automatically included.
|
|
|
|
*/
|
|
|
|
pa = DMAP_TO_PHYS((vm_offset_t)msgbufp->msg_ptr);
|
2011-01-21 10:26:26 +00:00
|
|
|
last_pa = pa + round_page(msgbufsize);
|
2010-05-16 19:25:56 +00:00
|
|
|
while (pa < last_pa) {
|
|
|
|
dump_add_page(pa);
|
|
|
|
pa += PAGE_SIZE;
|
|
|
|
}
|
Introduce minidumps. Full physical memory crash dumps are still available
via the debug.minidump sysctl and tunable.
Traditional dumps store all physical memory. This was once a good thing
when machines had a maximum of 64M of ram and 1GB of kvm. These days,
machines often have many gigabytes of ram and a smaller amount of kvm.
libkvm+kgdb don't have a way to access physical ram that is not mapped
into kvm at the time of the crash dump, so the extra ram being dumped
is mostly wasted.
Minidumps invert the process. Instead of dumping physical memory in
in order to guarantee that all of kvm's backing is dumped, minidumps
instead dump only memory that is actively mapped into kvm.
amd64 has a direct map region that things like UMA use. Obviously we
cannot dump all of the direct map region because that is effectively
an old style all-physical-memory dump. Instead, introduce a bitmap
and two helper routines (dump_add_page(pa) and dump_drop_page(pa)) that
allow certain critical direct map pages to be included in the dump.
uma_machdep.c's allocator is the intended consumer.
Dumps are a custom format. At the very beginning of the file is a header,
then a copy of the message buffer, then the bitmap of pages present in
the dump, then the final level of the kvm page table trees (2MB mappings
are expanded into a 4K page mappings), then the sparse physical pages
according to the bitmap. libkvm can now conveniently access the kvm
page table entries.
Booting my test 8GB machine, forcing it into ddb and forcing a dump
leads to a 48MB minidump. While this is a best case, I expect minidumps
to be in the 100MB-500MB range. Obviously, never larger than physical
memory of course.
minidumps are on by default. It would want be necessary to turn them off
if it was necessary to debug corrupt kernel page table management as that
would mess up minidumps as well.
Both minidumps and regular dumps are supported on the same machine.
2006-04-21 04:24:50 +00:00
|
|
|
#endif
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Compute the number of pages of memory that will be available for
|
2017-02-04 05:23:10 +00:00
|
|
|
* use, taking into account the overhead of a page structure per page.
|
|
|
|
* In other words, solve
|
|
|
|
* "available physical memory" - round_page(page_range *
|
|
|
|
* sizeof(struct vm_page)) = page_range * PAGE_SIZE
|
|
|
|
* for page_range.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2017-02-04 05:23:10 +00:00
|
|
|
low_avail = phys_avail[0];
|
|
|
|
high_avail = phys_avail[1];
|
2014-11-15 23:40:44 +00:00
|
|
|
for (i = 0; i < vm_phys_nsegs; i++) {
|
2017-02-04 05:23:10 +00:00
|
|
|
if (vm_phys_segs[i].start < low_avail)
|
|
|
|
low_avail = vm_phys_segs[i].start;
|
|
|
|
if (vm_phys_segs[i].end > high_avail)
|
|
|
|
high_avail = vm_phys_segs[i].end;
|
|
|
|
}
|
|
|
|
/* Skip the first chunk. It is already accounted for. */
|
|
|
|
for (i = 2; phys_avail[i + 1] != 0; i += 2) {
|
|
|
|
if (phys_avail[i] < low_avail)
|
|
|
|
low_avail = phys_avail[i];
|
|
|
|
if (phys_avail[i + 1] > high_avail)
|
|
|
|
high_avail = phys_avail[i + 1];
|
2014-11-15 23:40:44 +00:00
|
|
|
}
|
2017-02-04 05:23:10 +00:00
|
|
|
first_page = low_avail / PAGE_SIZE;
|
|
|
|
#ifdef VM_PHYSSEG_SPARSE
|
|
|
|
size = 0;
|
|
|
|
for (i = 0; i < vm_phys_nsegs; i++)
|
|
|
|
size += vm_phys_segs[i].end - vm_phys_segs[i].start;
|
2007-05-05 19:50:28 +00:00
|
|
|
for (i = 0; phys_avail[i + 1] != 0; i += 2)
|
2017-02-04 05:23:10 +00:00
|
|
|
size += phys_avail[i + 1] - phys_avail[i];
|
2007-05-05 19:50:28 +00:00
|
|
|
#elif defined(VM_PHYSSEG_DENSE)
|
2017-06-08 16:18:41 +00:00
|
|
|
size = high_avail - low_avail;
|
|
|
|
#else
|
|
|
|
#error "Either VM_PHYSSEG_DENSE or VM_PHYSSEG_SPARSE must be defined."
|
|
|
|
#endif
|
|
|
|
|
2019-08-18 23:07:56 +00:00
|
|
|
#ifdef PMAP_HAS_PAGE_ARRAY
|
|
|
|
pmap_page_array_startup(size / PAGE_SIZE);
|
|
|
|
biggestone = vm_phys_avail_largest();
|
|
|
|
end = new_end = phys_avail[biggestone + 1];
|
|
|
|
#else
|
2017-06-08 16:18:41 +00:00
|
|
|
#ifdef VM_PHYSSEG_DENSE
|
2017-02-04 05:23:10 +00:00
|
|
|
/*
|
|
|
|
* In the VM_PHYSSEG_DENSE case, the number of pages can account for
|
|
|
|
* the overhead of a page structure per page only if vm_page_array is
|
|
|
|
* allocated from the last physical memory chunk. Otherwise, we must
|
|
|
|
* allocate page structures representing the physical memory
|
|
|
|
* underlying vm_page_array, even though they will not be used.
|
|
|
|
*/
|
2017-06-08 16:18:41 +00:00
|
|
|
if (new_end != high_avail)
|
|
|
|
page_range = size / PAGE_SIZE;
|
2017-02-04 05:23:10 +00:00
|
|
|
else
|
2007-05-05 19:50:28 +00:00
|
|
|
#endif
|
2017-06-08 16:18:41 +00:00
|
|
|
{
|
|
|
|
page_range = size / (PAGE_SIZE + sizeof(struct vm_page));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the partial bytes remaining are large enough for
|
|
|
|
* a page (PAGE_SIZE) without a corresponding
|
|
|
|
* 'struct vm_page', then new_end will contain an
|
|
|
|
* extra page after subtracting the length of the VM
|
|
|
|
* page array. Compensate by subtracting an extra
|
|
|
|
* page from new_end.
|
|
|
|
*/
|
|
|
|
if (size % (PAGE_SIZE + sizeof(struct vm_page)) >= PAGE_SIZE) {
|
|
|
|
if (new_end == high_avail)
|
|
|
|
high_avail -= PAGE_SIZE;
|
|
|
|
new_end -= PAGE_SIZE;
|
|
|
|
}
|
|
|
|
}
|
2001-03-01 19:21:24 +00:00
|
|
|
end = new_end;
|
2019-08-18 23:07:56 +00:00
|
|
|
new_end = vm_page_array_alloc(&vaddr, end, page_range);
|
|
|
|
#endif
|
2017-09-07 21:43:39 +00:00
|
|
|
|
2007-12-29 19:53:04 +00:00
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
/*
|
2017-02-04 05:23:10 +00:00
|
|
|
* Allocate physical memory for the reservation management system's
|
|
|
|
* data structures, and map it.
|
2007-12-29 19:53:04 +00:00
|
|
|
*/
|
2019-08-18 23:07:56 +00:00
|
|
|
new_end = vm_reserv_startup(&vaddr, new_end);
|
2007-12-29 19:53:04 +00:00
|
|
|
#endif
|
2022-01-06 00:42:40 +00:00
|
|
|
#if defined(__aarch64__) || defined(__amd64__) || \
|
2019-10-14 13:04:04 +00:00
|
|
|
defined(__riscv) || defined(__powerpc64__)
|
2006-05-31 22:55:23 +00:00
|
|
|
/*
|
2017-02-04 05:23:10 +00:00
|
|
|
* Include vm_page_array and vm_reserv_array in a crash dump.
|
2006-05-31 22:55:23 +00:00
|
|
|
*/
|
2017-02-04 05:23:10 +00:00
|
|
|
for (pa = new_end; pa < end; pa += PAGE_SIZE)
|
2006-05-31 22:55:23 +00:00
|
|
|
dump_add_page(pa);
|
2015-03-24 20:07:27 +00:00
|
|
|
#endif
|
2003-03-17 03:16:00 +00:00
|
|
|
phys_avail[biggestone + 1] = new_end;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2014-11-15 23:40:44 +00:00
|
|
|
/*
|
|
|
|
* Add physical memory segments corresponding to the available
|
|
|
|
* physical pages.
|
|
|
|
*/
|
|
|
|
for (i = 0; phys_avail[i + 1] != 0; i += 2)
|
2019-08-18 07:06:31 +00:00
|
|
|
if (vm_phys_avail_size(i) != 0)
|
|
|
|
vm_phys_add_seg(phys_avail[i], phys_avail[i + 1]);
|
2014-11-15 23:40:44 +00:00
|
|
|
|
1999-03-19 05:21:03 +00:00
|
|
|
/*
|
Enable the new physical memory allocator.
This allocator uses a binary buddy system with a twist. First and
foremost, this allocator is required to support the implementation of
superpages. As a side effect, it enables a more robust implementation
of contigmalloc(9). Moreover, this reimplementation of
contigmalloc(9) eliminates the acquisition of Giant by
contigmalloc(..., M_NOWAIT, ...).
The twist is that this allocator tries to reduce the number of TLB
misses incurred by accesses through a direct map to small, UMA-managed
objects and page table pages. Roughly speaking, the physical pages
that are allocated for such purposes are clustered together in the
physical address space. The performance benefits vary. In the most
extreme case, a uniprocessor kernel running on an Opteron, I measured
an 18% reduction in system time during a buildworld.
This allocator does not implement page coloring. The reason is that
superpages have much the same effect. The contiguous physical memory
allocation necessary for a superpage is inherently colored.
Finally, the one caveat is that this allocator does not effectively
support prezeroed pages. I hope this is temporary. On i386, this is
a slight pessimization. However, on amd64, the beneficial effects of
the direct-map optimization outweigh the ill effects. I speculate
that this is true in general of machines with a direct map.
Approved by: re
2007-06-16 04:57:06 +00:00
|
|
|
* Initialize the physical memory allocator.
|
|
|
|
*/
|
|
|
|
vm_phys_init();
|
|
|
|
|
|
|
|
/*
|
2017-09-07 21:43:39 +00:00
|
|
|
* Initialize the page structures and add every available page to the
|
|
|
|
* physical memory allocator's free lists.
|
1999-03-19 05:21:03 +00:00
|
|
|
*/
|
2018-07-05 16:43:15 +00:00
|
|
|
#if defined(__i386__) && defined(VM_PHYSSEG_DENSE)
|
|
|
|
for (ii = 0; ii < vm_page_array_size; ii++) {
|
|
|
|
m = &vm_page_array[ii];
|
|
|
|
vm_page_init_page(m, (first_page + ii) << PAGE_SHIFT, 0);
|
|
|
|
m->flags = PG_FICTITIOUS;
|
|
|
|
}
|
|
|
|
#endif
|
2014-03-22 10:26:09 +00:00
|
|
|
vm_cnt.v_page_count = 0;
|
2017-09-07 21:43:39 +00:00
|
|
|
for (segind = 0; segind < vm_phys_nsegs; segind++) {
|
|
|
|
seg = &vm_phys_segs[segind];
|
2017-11-27 17:46:38 +00:00
|
|
|
for (m = seg->first_page, pa = seg->start; pa < seg->end;
|
|
|
|
m++, pa += PAGE_SIZE)
|
|
|
|
vm_page_init_page(m, pa, segind);
|
2017-09-07 21:43:39 +00:00
|
|
|
|
|
|
|
/*
|
2021-09-14 12:25:37 +00:00
|
|
|
* Add the segment's pages that are covered by one of
|
|
|
|
* phys_avail's ranges to the free lists.
|
2017-09-07 21:43:39 +00:00
|
|
|
*/
|
|
|
|
for (i = 0; phys_avail[i + 1] != 0; i += 2) {
|
2021-09-17 18:48:42 +00:00
|
|
|
if (seg->end <= phys_avail[i] ||
|
2021-09-14 12:25:37 +00:00
|
|
|
seg->start >= phys_avail[i + 1])
|
2017-09-07 21:43:39 +00:00
|
|
|
continue;
|
|
|
|
|
2021-09-14 12:25:37 +00:00
|
|
|
startp = MAX(seg->start, phys_avail[i]);
|
|
|
|
endp = MIN(seg->end, phys_avail[i + 1]);
|
|
|
|
pagecount = (u_long)atop(endp - startp);
|
|
|
|
if (pagecount == 0)
|
|
|
|
continue;
|
2017-09-07 21:43:39 +00:00
|
|
|
|
2021-09-17 18:48:42 +00:00
|
|
|
m = seg->first_page + atop(startp - seg->start);
|
2018-02-06 22:10:07 +00:00
|
|
|
vmd = VM_DOMAIN(seg->domain);
|
|
|
|
vm_domain_free_lock(vmd);
|
2019-05-31 21:02:42 +00:00
|
|
|
vm_phys_enqueue_contig(m, pagecount);
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_domain_free_unlock(vmd);
|
2018-03-15 19:23:07 +00:00
|
|
|
vm_domain_freecnt_inc(vmd, pagecount);
|
2017-09-07 21:43:39 +00:00
|
|
|
vm_cnt.v_page_count += (u_int)pagecount;
|
|
|
|
vmd->vmd_page_count += (u_int)pagecount;
|
2021-09-17 18:48:42 +00:00
|
|
|
vmd->vmd_segs |= 1UL << segind;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
2015-04-29 15:57:14 +00:00
|
|
|
|
2017-09-07 21:43:39 +00:00
|
|
|
/*
|
2020-07-09 22:38:36 +00:00
|
|
|
* Remove blacklisted pages from the physical memory allocator.
|
2017-09-07 21:43:39 +00:00
|
|
|
*/
|
2020-07-09 22:38:36 +00:00
|
|
|
TAILQ_INIT(&blacklist_head);
|
|
|
|
vm_page_blacklist_load(&list, &listend);
|
|
|
|
vm_page_blacklist_check(list, listend);
|
2015-04-29 15:57:14 +00:00
|
|
|
|
2020-07-09 22:38:36 +00:00
|
|
|
list = kern_getenv("vm.blacklist");
|
|
|
|
vm_page_blacklist_check(list, NULL);
|
2015-04-29 15:57:14 +00:00
|
|
|
|
2006-06-23 16:44:24 +00:00
|
|
|
freeenv(list);
|
2007-12-29 19:53:04 +00:00
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
/*
|
|
|
|
* Initialize the reservation management system.
|
|
|
|
*/
|
|
|
|
vm_reserv_init();
|
|
|
|
#endif
|
2018-01-12 22:48:23 +00:00
|
|
|
|
2001-03-07 05:29:21 +00:00
|
|
|
return (vaddr);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2011-09-06 10:30:11 +00:00
|
|
|
void
|
|
|
|
vm_page_reference(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
|
|
|
vm_page_aflag_set(m, PGA_REFERENCED);
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
2020-02-27 02:37:27 +00:00
|
|
|
/*
|
|
|
|
* vm_page_trybusy
|
|
|
|
*
|
|
|
|
* Helper routine for grab functions to trylock busy.
|
|
|
|
*
|
|
|
|
* Returns true on success and false on failure.
|
|
|
|
*/
|
2019-12-08 01:16:22 +00:00
|
|
|
static bool
|
2020-02-27 02:37:27 +00:00
|
|
|
vm_page_trybusy(vm_page_t m, int allocflags)
|
2019-12-08 01:16:22 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
if ((allocflags & (VM_ALLOC_SBUSY | VM_ALLOC_IGN_SBUSY)) != 0)
|
2020-02-27 02:37:27 +00:00
|
|
|
return (vm_page_trysbusy(m));
|
2019-12-08 01:16:22 +00:00
|
|
|
else
|
2020-02-27 02:37:27 +00:00
|
|
|
return (vm_page_tryxbusy(m));
|
2019-12-08 01:16:22 +00:00
|
|
|
}
|
|
|
|
|
2019-12-24 18:38:06 +00:00
|
|
|
/*
|
2020-02-27 02:37:27 +00:00
|
|
|
* vm_page_tryacquire
|
2019-12-24 18:38:06 +00:00
|
|
|
*
|
2020-02-27 02:37:27 +00:00
|
|
|
* Helper routine for grab functions to trylock busy and wire.
|
|
|
|
*
|
|
|
|
* Returns true on success and false on failure.
|
2019-12-24 18:38:06 +00:00
|
|
|
*/
|
2020-02-27 02:37:27 +00:00
|
|
|
static inline bool
|
|
|
|
vm_page_tryacquire(vm_page_t m, int allocflags)
|
2019-12-08 01:16:22 +00:00
|
|
|
{
|
2020-02-27 02:37:27 +00:00
|
|
|
bool locked;
|
2019-12-08 01:16:22 +00:00
|
|
|
|
2020-02-27 02:37:27 +00:00
|
|
|
locked = vm_page_trybusy(m, allocflags);
|
|
|
|
if (locked && (allocflags & VM_ALLOC_WIRED) != 0)
|
|
|
|
vm_page_wire(m);
|
|
|
|
return (locked);
|
2019-12-08 01:16:22 +00:00
|
|
|
}
|
|
|
|
|
2019-09-10 19:08:01 +00:00
|
|
|
/*
|
|
|
|
* vm_page_busy_acquire:
|
|
|
|
*
|
|
|
|
* Acquire the busy lock as described by VM_ALLOC_* flags. Will loop
|
|
|
|
* and drop the object lock if necessary.
|
|
|
|
*/
|
2019-12-08 01:16:22 +00:00
|
|
|
bool
|
2019-09-10 19:08:01 +00:00
|
|
|
vm_page_busy_acquire(vm_page_t m, int allocflags)
|
|
|
|
{
|
|
|
|
vm_object_t obj;
|
|
|
|
bool locked;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The page-specific object must be cached because page
|
|
|
|
* identity can change during the sleep, causing the
|
|
|
|
* re-lock of a different object.
|
|
|
|
* It is assumed that a reference to the object is already
|
|
|
|
* held by the callers.
|
|
|
|
*/
|
2020-12-27 21:50:54 +00:00
|
|
|
obj = atomic_load_ptr(&m->object);
|
2019-09-10 19:08:01 +00:00
|
|
|
for (;;) {
|
2020-02-27 02:37:27 +00:00
|
|
|
if (vm_page_tryacquire(m, allocflags))
|
2019-12-08 01:16:22 +00:00
|
|
|
return (true);
|
2019-09-10 19:08:01 +00:00
|
|
|
if ((allocflags & VM_ALLOC_NOWAIT) != 0)
|
2019-12-08 01:16:22 +00:00
|
|
|
return (false);
|
2019-10-15 03:41:36 +00:00
|
|
|
if (obj != NULL)
|
2019-09-10 19:08:01 +00:00
|
|
|
locked = VM_OBJECT_WOWNED(obj);
|
2019-10-15 03:41:36 +00:00
|
|
|
else
|
2019-12-08 01:16:22 +00:00
|
|
|
locked = false;
|
2019-10-15 03:41:36 +00:00
|
|
|
MPASS(locked || vm_page_wired(m));
|
2020-02-17 01:08:00 +00:00
|
|
|
if (_vm_page_busy_sleep(obj, m, m->pindex, "vmpba", allocflags,
|
|
|
|
locked) && locked)
|
2019-09-10 19:08:01 +00:00
|
|
|
VM_OBJECT_WLOCK(obj);
|
|
|
|
if ((allocflags & VM_ALLOC_WAITFAIL) != 0)
|
2019-12-08 01:16:22 +00:00
|
|
|
return (false);
|
2019-10-23 17:58:19 +00:00
|
|
|
KASSERT(m->object == obj || m->object == NULL,
|
|
|
|
("vm_page_busy_acquire: page %p does not belong to %p",
|
|
|
|
m, obj));
|
2019-09-10 19:08:01 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-08-09 11:11:11 +00:00
|
|
|
/*
|
|
|
|
* vm_page_busy_downgrade:
|
|
|
|
*
|
|
|
|
* Downgrade an exclusive busy page into a single shared busy page.
|
|
|
|
*/
|
2001-07-04 20:15:18 +00:00
|
|
|
void
|
2013-08-09 11:11:11 +00:00
|
|
|
vm_page_busy_downgrade(vm_page_t m)
|
2001-07-04 20:15:18 +00:00
|
|
|
{
|
2013-08-09 11:11:11 +00:00
|
|
|
u_int x;
|
2004-10-24 23:53:47 +00:00
|
|
|
|
2013-08-09 11:11:11 +00:00
|
|
|
vm_page_assert_xbusied(m);
|
|
|
|
|
2020-07-29 19:38:49 +00:00
|
|
|
x = vm_page_busy_fetch(m);
|
2013-08-09 11:11:11 +00:00
|
|
|
for (;;) {
|
2019-09-10 18:27:45 +00:00
|
|
|
if (atomic_fcmpset_rel_int(&m->busy_lock,
|
|
|
|
&x, VPB_SHARERS_WORD(1)))
|
2013-08-09 11:11:11 +00:00
|
|
|
break;
|
2016-10-11 18:09:37 +00:00
|
|
|
}
|
2019-09-10 18:27:45 +00:00
|
|
|
if ((x & VPB_BIT_WAITERS) != 0)
|
2016-10-11 18:09:37 +00:00
|
|
|
wakeup(m);
|
2013-08-09 11:11:11 +00:00
|
|
|
}
|
|
|
|
|
2019-10-15 03:35:11 +00:00
|
|
|
/*
|
|
|
|
*
|
|
|
|
* vm_page_busy_tryupgrade:
|
|
|
|
*
|
|
|
|
* Attempt to upgrade a single shared busy into an exclusive busy.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_busy_tryupgrade(vm_page_t m)
|
|
|
|
{
|
2019-11-24 19:12:23 +00:00
|
|
|
u_int ce, x;
|
2019-10-15 03:35:11 +00:00
|
|
|
|
|
|
|
vm_page_assert_sbusied(m);
|
|
|
|
|
2020-07-29 19:38:49 +00:00
|
|
|
x = vm_page_busy_fetch(m);
|
2019-11-24 19:12:23 +00:00
|
|
|
ce = VPB_CURTHREAD_EXCLUSIVE;
|
2019-10-15 03:35:11 +00:00
|
|
|
for (;;) {
|
|
|
|
if (VPB_SHARERS(x) > 1)
|
|
|
|
return (0);
|
|
|
|
KASSERT((x & ~VPB_BIT_WAITERS) == VPB_SHARERS_WORD(1),
|
|
|
|
("vm_page_busy_tryupgrade: invalid lock state"));
|
|
|
|
if (!atomic_fcmpset_acq_int(&m->busy_lock, &x,
|
2019-11-24 19:12:23 +00:00
|
|
|
ce | (x & VPB_BIT_WAITERS)))
|
2019-10-15 03:35:11 +00:00
|
|
|
continue;
|
|
|
|
return (1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-08-09 11:11:11 +00:00
|
|
|
/*
|
|
|
|
* vm_page_sbusied:
|
|
|
|
*
|
|
|
|
* Return a positive value if the page is shared busied, 0 otherwise.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_sbusied(vm_page_t m)
|
|
|
|
{
|
|
|
|
u_int x;
|
|
|
|
|
2020-07-29 19:38:49 +00:00
|
|
|
x = vm_page_busy_fetch(m);
|
2013-08-09 11:11:11 +00:00
|
|
|
return ((x & VPB_BIT_SHARED) != 0 && x != VPB_UNBUSIED);
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2013-08-09 11:11:11 +00:00
|
|
|
* vm_page_sunbusy:
|
2001-07-04 20:15:18 +00:00
|
|
|
*
|
2013-08-09 11:11:11 +00:00
|
|
|
* Shared unbusy a page.
|
2001-07-04 20:15:18 +00:00
|
|
|
*/
|
|
|
|
void
|
2013-08-09 11:11:11 +00:00
|
|
|
vm_page_sunbusy(vm_page_t m)
|
2001-07-04 20:15:18 +00:00
|
|
|
{
|
2013-08-09 11:11:11 +00:00
|
|
|
u_int x;
|
2004-10-25 19:52:44 +00:00
|
|
|
|
2022-08-17 16:24:14 +00:00
|
|
|
vm_page_assert_sbusied(m);
|
|
|
|
|
|
|
|
x = vm_page_busy_fetch(m);
|
|
|
|
for (;;) {
|
|
|
|
KASSERT(x != VPB_FREED,
|
|
|
|
("vm_page_sunbusy: Unlocking freed page."));
|
|
|
|
if (VPB_SHARERS(x) > 1) {
|
|
|
|
if (atomic_fcmpset_int(&m->busy_lock, &x,
|
|
|
|
x - VPB_ONE_SHARER))
|
|
|
|
break;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
KASSERT((x & ~VPB_BIT_WAITERS) == VPB_SHARERS_WORD(1),
|
|
|
|
("vm_page_sunbusy: invalid lock state"));
|
|
|
|
if (!atomic_fcmpset_rel_int(&m->busy_lock, &x, VPB_UNBUSIED))
|
|
|
|
continue;
|
|
|
|
if ((x & VPB_BIT_WAITERS) == 0)
|
|
|
|
break;
|
2001-07-04 20:15:18 +00:00
|
|
|
wakeup(m);
|
2022-08-17 16:24:14 +00:00
|
|
|
break;
|
|
|
|
}
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2013-08-09 11:11:11 +00:00
|
|
|
* vm_page_busy_sleep:
|
2001-07-04 20:15:18 +00:00
|
|
|
*
|
2019-09-10 18:27:45 +00:00
|
|
|
* Sleep if the page is busy, using the page pointer as wchan.
|
2021-11-15 16:35:44 +00:00
|
|
|
* This is used to implement the hard-path of the busying mechanism.
|
|
|
|
*
|
|
|
|
* If VM_ALLOC_IGN_SBUSY is specified in allocflags, the function
|
|
|
|
* will not sleep if the page is shared-busy.
|
2001-07-04 20:15:18 +00:00
|
|
|
*
|
2021-11-15 16:35:44 +00:00
|
|
|
* The object lock must be held on entry.
|
2019-09-10 18:27:45 +00:00
|
|
|
*
|
2021-11-15 16:35:44 +00:00
|
|
|
* Returns true if it slept and dropped the object lock, or false
|
|
|
|
* if there was no sleep and the lock is still held.
|
2001-07-04 20:15:18 +00:00
|
|
|
*/
|
2021-11-15 16:35:44 +00:00
|
|
|
bool
|
|
|
|
vm_page_busy_sleep(vm_page_t m, const char *wmesg, int allocflags)
|
2001-07-04 20:15:18 +00:00
|
|
|
{
|
2019-09-10 18:27:45 +00:00
|
|
|
vm_object_t obj;
|
2004-10-24 23:53:47 +00:00
|
|
|
|
2019-09-10 18:27:45 +00:00
|
|
|
obj = m->object;
|
|
|
|
VM_OBJECT_ASSERT_LOCKED(obj);
|
2013-08-09 11:11:11 +00:00
|
|
|
|
2021-11-15 16:35:44 +00:00
|
|
|
return (_vm_page_busy_sleep(obj, m, m->pindex, wmesg, allocflags,
|
|
|
|
true));
|
2019-10-15 03:41:36 +00:00
|
|
|
}
|
|
|
|
|
2020-02-17 01:08:00 +00:00
|
|
|
/*
|
|
|
|
* vm_page_busy_sleep_unlocked:
|
|
|
|
*
|
|
|
|
* Sleep if the page is busy, using the page pointer as wchan.
|
|
|
|
* This is used to implement the hard-path of busying mechanism.
|
|
|
|
*
|
2021-11-15 16:35:44 +00:00
|
|
|
* If VM_ALLOC_IGN_SBUSY is specified in allocflags, the function
|
|
|
|
* will not sleep if the page is shared-busy.
|
2020-02-17 01:08:00 +00:00
|
|
|
*
|
|
|
|
* The object lock must not be held on entry. The operation will
|
|
|
|
* return if the page changes identity.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_busy_sleep_unlocked(vm_object_t obj, vm_page_t m, vm_pindex_t pindex,
|
2021-11-15 16:35:44 +00:00
|
|
|
const char *wmesg, int allocflags)
|
2020-02-17 01:08:00 +00:00
|
|
|
{
|
|
|
|
VM_OBJECT_ASSERT_UNLOCKED(obj);
|
|
|
|
|
2021-11-15 16:35:44 +00:00
|
|
|
(void)_vm_page_busy_sleep(obj, m, pindex, wmesg, allocflags, false);
|
2020-02-17 01:08:00 +00:00
|
|
|
}
|
|
|
|
|
2019-12-24 18:38:06 +00:00
|
|
|
/*
|
|
|
|
* _vm_page_busy_sleep:
|
|
|
|
*
|
2020-02-17 01:08:00 +00:00
|
|
|
* Internal busy sleep function. Verifies the page identity and
|
|
|
|
* lockstate against parameters. Returns true if it sleeps and
|
|
|
|
* false otherwise.
|
|
|
|
*
|
2021-11-15 16:35:44 +00:00
|
|
|
* allocflags uses VM_ALLOC_* flags to specify the lock required.
|
|
|
|
*
|
2020-02-17 01:08:00 +00:00
|
|
|
* If locked is true the lock will be dropped for any true returns
|
|
|
|
* and held for any false returns.
|
2019-12-24 18:38:06 +00:00
|
|
|
*/
|
|
|
|
static bool
|
2020-02-17 01:08:00 +00:00
|
|
|
_vm_page_busy_sleep(vm_object_t obj, vm_page_t m, vm_pindex_t pindex,
|
|
|
|
const char *wmesg, int allocflags, bool locked)
|
2019-10-15 03:41:36 +00:00
|
|
|
{
|
2020-02-17 01:08:00 +00:00
|
|
|
bool xsleep;
|
2019-10-15 03:41:36 +00:00
|
|
|
u_int x;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the object is busy we must wait for that to drain to zero
|
|
|
|
* before trying the page again.
|
|
|
|
*/
|
|
|
|
if (obj != NULL && vm_object_busied(obj)) {
|
|
|
|
if (locked)
|
|
|
|
VM_OBJECT_DROP(obj);
|
|
|
|
vm_object_busy_wait(obj, wmesg);
|
2020-02-17 01:08:00 +00:00
|
|
|
return (true);
|
2019-10-15 03:41:36 +00:00
|
|
|
}
|
2020-02-17 01:08:00 +00:00
|
|
|
|
|
|
|
if (!vm_page_busied(m))
|
2019-12-24 18:38:06 +00:00
|
|
|
return (false);
|
2020-02-17 01:08:00 +00:00
|
|
|
|
|
|
|
xsleep = (allocflags & (VM_ALLOC_SBUSY | VM_ALLOC_IGN_SBUSY)) != 0;
|
|
|
|
sleepq_lock(m);
|
2020-07-29 19:38:49 +00:00
|
|
|
x = vm_page_busy_fetch(m);
|
2020-02-17 01:08:00 +00:00
|
|
|
do {
|
|
|
|
/*
|
|
|
|
* If the page changes objects or becomes unlocked we can
|
|
|
|
* simply return.
|
|
|
|
*/
|
|
|
|
if (x == VPB_UNBUSIED ||
|
|
|
|
(xsleep && (x & VPB_BIT_SHARED) != 0) ||
|
|
|
|
m->object != obj || m->pindex != pindex) {
|
|
|
|
sleepq_release(m);
|
|
|
|
return (false);
|
|
|
|
}
|
|
|
|
if ((x & VPB_BIT_WAITERS) != 0)
|
|
|
|
break;
|
|
|
|
} while (!atomic_fcmpset_int(&m->busy_lock, &x, x | VPB_BIT_WAITERS));
|
2019-10-15 03:41:36 +00:00
|
|
|
if (locked)
|
|
|
|
VM_OBJECT_DROP(obj);
|
2019-11-07 18:26:29 +00:00
|
|
|
DROP_GIANT();
|
2019-09-10 18:27:45 +00:00
|
|
|
sleepq_add(m, NULL, wmesg, 0, 0);
|
|
|
|
sleepq_wait(m, PVM);
|
2019-11-07 18:26:29 +00:00
|
|
|
PICKUP_GIANT();
|
2020-02-17 01:08:00 +00:00
|
|
|
return (true);
|
2013-08-09 11:11:11 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_trysbusy:
|
|
|
|
*
|
|
|
|
* Try to shared busy a page.
|
|
|
|
* If the operation succeeds 1 is returned otherwise 0.
|
|
|
|
* The operation never sleeps.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_trysbusy(vm_page_t m)
|
|
|
|
{
|
2019-10-15 03:41:36 +00:00
|
|
|
vm_object_t obj;
|
2013-08-09 11:11:11 +00:00
|
|
|
u_int x;
|
|
|
|
|
2019-10-15 03:41:36 +00:00
|
|
|
obj = m->object;
|
2020-07-29 19:38:49 +00:00
|
|
|
x = vm_page_busy_fetch(m);
|
2013-09-05 12:54:40 +00:00
|
|
|
for (;;) {
|
|
|
|
if ((x & VPB_BIT_SHARED) == 0)
|
|
|
|
return (0);
|
2019-10-15 03:41:36 +00:00
|
|
|
/*
|
|
|
|
* Reduce the window for transient busies that will trigger
|
|
|
|
* false negatives in vm_page_ps_test().
|
|
|
|
*/
|
|
|
|
if (obj != NULL && vm_object_busied(obj))
|
|
|
|
return (0);
|
2019-09-10 18:27:45 +00:00
|
|
|
if (atomic_fcmpset_acq_int(&m->busy_lock, &x,
|
|
|
|
x + VPB_ONE_SHARER))
|
2019-10-15 03:41:36 +00:00
|
|
|
break;
|
2013-09-05 12:54:40 +00:00
|
|
|
}
|
2019-10-15 03:41:36 +00:00
|
|
|
|
|
|
|
/* Refetch the object now that we're guaranteed that it is stable. */
|
|
|
|
obj = m->object;
|
|
|
|
if (obj != NULL && vm_object_busied(obj)) {
|
|
|
|
vm_page_sunbusy(m);
|
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
return (1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_tryxbusy:
|
|
|
|
*
|
|
|
|
* Try to exclusive busy a page.
|
|
|
|
* If the operation succeeds 1 is returned otherwise 0.
|
|
|
|
* The operation never sleeps.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_tryxbusy(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_object_t obj;
|
|
|
|
|
2020-07-29 19:38:49 +00:00
|
|
|
if (atomic_cmpset_acq_int(&m->busy_lock, VPB_UNBUSIED,
|
2019-11-24 19:12:23 +00:00
|
|
|
VPB_CURTHREAD_EXCLUSIVE) == 0)
|
2019-10-15 03:41:36 +00:00
|
|
|
return (0);
|
|
|
|
|
|
|
|
obj = m->object;
|
|
|
|
if (obj != NULL && vm_object_busied(obj)) {
|
|
|
|
vm_page_xunbusy(m);
|
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
return (1);
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
2019-11-24 19:12:23 +00:00
|
|
|
static void
|
|
|
|
vm_page_xunbusy_hard_tail(vm_page_t m)
|
|
|
|
{
|
|
|
|
atomic_store_rel_int(&m->busy_lock, VPB_UNBUSIED);
|
|
|
|
/* Wake the waiter. */
|
|
|
|
wakeup(m);
|
|
|
|
}
|
|
|
|
|
2013-08-09 11:11:11 +00:00
|
|
|
/*
|
|
|
|
* vm_page_xunbusy_hard:
|
|
|
|
*
|
2019-09-10 18:27:45 +00:00
|
|
|
* Called when unbusy has failed because there is a waiter.
|
2013-08-09 11:11:11 +00:00
|
|
|
*/
|
2001-07-04 20:15:18 +00:00
|
|
|
void
|
2013-08-09 11:11:11 +00:00
|
|
|
vm_page_xunbusy_hard(vm_page_t m)
|
2001-07-04 20:15:18 +00:00
|
|
|
{
|
2013-08-09 11:11:11 +00:00
|
|
|
vm_page_assert_xbusied(m);
|
2019-11-24 19:12:23 +00:00
|
|
|
vm_page_xunbusy_hard_tail(m);
|
|
|
|
}
|
2013-08-09 11:11:11 +00:00
|
|
|
|
2019-11-24 19:12:23 +00:00
|
|
|
void
|
|
|
|
vm_page_xunbusy_hard_unchecked(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_page_assert_xbusied_unchecked(m);
|
|
|
|
vm_page_xunbusy_hard_tail(m);
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
2020-02-04 20:33:01 +00:00
|
|
|
static void
|
|
|
|
vm_page_busy_free(vm_page_t m)
|
|
|
|
{
|
|
|
|
u_int x;
|
|
|
|
|
|
|
|
atomic_thread_fence_rel();
|
|
|
|
x = atomic_swap_int(&m->busy_lock, VPB_FREED);
|
|
|
|
if ((x & VPB_BIT_WAITERS) != 0)
|
|
|
|
wakeup(m);
|
|
|
|
}
|
|
|
|
|
2010-12-17 22:41:22 +00:00
|
|
|
/*
|
|
|
|
* vm_page_unhold_pages:
|
|
|
|
*
|
|
|
|
* Unhold each of the pages that is referenced by the given array.
|
2015-03-24 20:07:27 +00:00
|
|
|
*/
|
2010-12-17 22:41:22 +00:00
|
|
|
void
|
|
|
|
vm_page_unhold_pages(vm_page_t *ma, int count)
|
|
|
|
{
|
|
|
|
|
|
|
|
for (; count != 0; count--) {
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
vm_page_unwire(*ma, PQ_ACTIVE);
|
2010-12-17 22:41:22 +00:00
|
|
|
ma++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-05-12 20:42:56 +00:00
|
|
|
vm_page_t
|
|
|
|
PHYS_TO_VM_PAGE(vm_paddr_t pa)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
|
|
|
#ifdef VM_PHYSSEG_SPARSE
|
|
|
|
m = vm_phys_paddr_to_vm_page(pa);
|
|
|
|
if (m == NULL)
|
|
|
|
m = vm_phys_fictitious_to_vm_page(pa);
|
|
|
|
return (m);
|
|
|
|
#elif defined(VM_PHYSSEG_DENSE)
|
|
|
|
long pi;
|
|
|
|
|
|
|
|
pi = atop(pa);
|
2012-05-22 07:04:23 +00:00
|
|
|
if (pi >= first_page && (pi - first_page) < vm_page_array_size) {
|
2012-05-12 20:42:56 +00:00
|
|
|
m = &vm_page_array[pi - first_page];
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
return (vm_phys_fictitious_to_vm_page(pa));
|
|
|
|
#else
|
|
|
|
#error "Either VM_PHYSSEG_DENSE or VM_PHYSSEG_SPARSE must be defined."
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2011-03-11 07:07:48 +00:00
|
|
|
/*
|
|
|
|
* vm_page_getfake:
|
|
|
|
*
|
|
|
|
* Create a fictitious page with the specified physical address and
|
|
|
|
* memory attribute. The memory attribute is the only the machine-
|
|
|
|
* dependent aspect of a fictitious page that must be initialized.
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_getfake(vm_paddr_t paddr, vm_memattr_t memattr)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
|
|
|
m = uma_zalloc(fakepg_zone, M_WAITOK | M_ZERO);
|
2012-05-12 20:34:22 +00:00
|
|
|
vm_page_initfake(m, paddr, memattr);
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_initfake(vm_page_t m, vm_paddr_t paddr, vm_memattr_t memattr)
|
|
|
|
{
|
|
|
|
|
|
|
|
if ((m->flags & PG_FICTITIOUS) != 0) {
|
|
|
|
/*
|
|
|
|
* The page's memattr might have changed since the
|
|
|
|
* previous initialization. Update the pmap to the
|
|
|
|
* new memattr.
|
|
|
|
*/
|
|
|
|
goto memattr;
|
|
|
|
}
|
2011-03-11 07:07:48 +00:00
|
|
|
m->phys_addr = paddr;
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.queue = PQ_NONE;
|
2011-03-11 07:07:48 +00:00
|
|
|
/* Fictitious pages don't use "segind". */
|
|
|
|
m->flags = PG_FICTITIOUS;
|
|
|
|
/* Fictitious pages don't use "order" or "pool". */
|
2013-08-09 11:11:11 +00:00
|
|
|
m->oflags = VPO_UNMANAGED;
|
2019-11-24 19:12:23 +00:00
|
|
|
m->busy_lock = VPB_CURTHREAD_EXCLUSIVE;
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
/* Fictitious pages are unevictable. */
|
|
|
|
m->ref_count = 1;
|
2013-07-03 23:38:37 +00:00
|
|
|
pmap_page_init(m);
|
2012-05-12 20:34:22 +00:00
|
|
|
memattr:
|
2011-03-11 07:07:48 +00:00
|
|
|
pmap_page_set_memattr(m, memattr);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_putfake:
|
|
|
|
*
|
|
|
|
* Release a fictitious page.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_putfake(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
2012-05-12 20:27:51 +00:00
|
|
|
KASSERT((m->oflags & VPO_UNMANAGED) != 0, ("managed %p", m));
|
2011-03-11 07:07:48 +00:00
|
|
|
KASSERT((m->flags & PG_FICTITIOUS) != 0,
|
|
|
|
("vm_page_putfake: bad page %p", m));
|
2020-02-04 20:33:01 +00:00
|
|
|
vm_page_assert_xbusied(m);
|
|
|
|
vm_page_busy_free(m);
|
2011-03-11 07:07:48 +00:00
|
|
|
uma_zfree(fakepg_zone, m);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_updatefake:
|
|
|
|
*
|
|
|
|
* Update the given fictitious page to the specified physical address and
|
|
|
|
* memory attribute.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_updatefake(vm_page_t m, vm_paddr_t paddr, vm_memattr_t memattr)
|
|
|
|
{
|
|
|
|
|
|
|
|
KASSERT((m->flags & PG_FICTITIOUS) != 0,
|
|
|
|
("vm_page_updatefake: bad page %p", m));
|
|
|
|
m->phys_addr = paddr;
|
|
|
|
pmap_page_set_memattr(m, memattr);
|
|
|
|
}
|
|
|
|
|
2001-07-04 20:15:18 +00:00
|
|
|
/*
|
|
|
|
* vm_page_free:
|
|
|
|
*
|
2007-02-17 19:37:00 +00:00
|
|
|
* Free a page.
|
2001-07-04 20:15:18 +00:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_free(vm_page_t m)
|
|
|
|
{
|
2007-02-18 05:54:42 +00:00
|
|
|
|
|
|
|
m->flags &= ~PG_ZERO;
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_free_toq(m);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_free_zero:
|
|
|
|
*
|
|
|
|
* Free a page to the zerod-pages queue
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_free_zero(vm_page_t m)
|
|
|
|
{
|
2007-02-18 05:54:42 +00:00
|
|
|
|
|
|
|
m->flags |= PG_ZERO;
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_free_toq(m);
|
|
|
|
}
|
|
|
|
|
2012-08-04 18:16:43 +00:00
|
|
|
/*
|
2016-08-30 05:56:21 +00:00
|
|
|
* Unbusy and handle the page queueing for a page from a getpages request that
|
|
|
|
* was optionally read ahead or behind.
|
2012-08-04 18:16:43 +00:00
|
|
|
*/
|
|
|
|
void
|
2012-08-14 11:45:47 +00:00
|
|
|
vm_page_readahead_finish(vm_page_t m)
|
2012-08-04 18:16:43 +00:00
|
|
|
{
|
|
|
|
|
A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().
o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.
Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.
Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().
o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.
This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.
Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
2015-12-16 21:30:45 +00:00
|
|
|
/* We shouldn't put invalid pages on queues. */
|
2019-10-15 03:45:41 +00:00
|
|
|
KASSERT(!vm_page_none_valid(m), ("%s: %p is invalid", __func__, m));
|
A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().
o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.
Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.
Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().
o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.
This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.
Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
2015-12-16 21:30:45 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Since the page is not the actually needed one, whether it should
|
|
|
|
* be activated or deactivated is not obvious. Empirical results
|
|
|
|
* have shown that deactivating the page is usually the best choice,
|
|
|
|
* unless the page is wanted by another thread.
|
|
|
|
*/
|
2020-07-29 19:38:49 +00:00
|
|
|
if ((vm_page_busy_fetch(m) & VPB_BIT_WAITERS) != 0)
|
A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().
o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.
Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.
Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().
o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.
This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.
Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
2015-12-16 21:30:45 +00:00
|
|
|
vm_page_activate(m);
|
|
|
|
else
|
|
|
|
vm_page_deactivate(m);
|
2019-11-24 19:12:23 +00:00
|
|
|
vm_page_xunbusy_unchecked(m);
|
2012-08-04 18:16:43 +00:00
|
|
|
}
|
|
|
|
|
2020-07-17 23:09:36 +00:00
|
|
|
/*
|
|
|
|
* Destroy the identity of an invalid page and free it if possible.
|
|
|
|
* This is intended to be used when reading a page from backing store fails.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_free_invalid(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
|
|
|
KASSERT(vm_page_none_valid(m), ("page %p is valid", m));
|
|
|
|
KASSERT(!pmap_page_is_mapped(m), ("page %p is mapped", m));
|
|
|
|
KASSERT(m->object != NULL, ("page %p has no object", m));
|
|
|
|
VM_OBJECT_ASSERT_WLOCKED(m->object);
|
|
|
|
|
2020-07-27 14:25:10 +00:00
|
|
|
/*
|
|
|
|
* We may be attempting to free the page as part of the handling for an
|
|
|
|
* I/O error, in which case the page was xbusied by a different thread.
|
|
|
|
*/
|
|
|
|
vm_page_xbusy_claim(m);
|
|
|
|
|
2020-07-17 23:09:36 +00:00
|
|
|
/*
|
|
|
|
* If someone has wired this page while the object lock
|
|
|
|
* was not held, then the thread that unwires is responsible
|
|
|
|
* for freeing the page. Otherwise just free the page now.
|
|
|
|
* The wire count of this unmapped page cannot change while
|
|
|
|
* we have the page xbusy and the page's object wlocked.
|
|
|
|
*/
|
|
|
|
if (vm_page_remove(m))
|
|
|
|
vm_page_free(m);
|
|
|
|
}
|
|
|
|
|
2001-07-04 20:15:18 +00:00
|
|
|
/*
|
2012-06-20 23:25:47 +00:00
|
|
|
* vm_page_dirty_KBI: [ internal use only ]
|
2001-07-04 20:15:18 +00:00
|
|
|
*
|
2011-06-19 19:13:24 +00:00
|
|
|
* Set all bits in the page's dirty field.
|
|
|
|
*
|
2011-09-28 14:57:50 +00:00
|
|
|
* The object containing the specified page must be locked if the
|
|
|
|
* call is made from the machine-independent layer.
|
|
|
|
*
|
2011-06-19 19:13:24 +00:00
|
|
|
* See vm_page_clear_dirty_mask().
|
2012-06-20 23:25:47 +00:00
|
|
|
*
|
|
|
|
* This function should only be called by vm_page_dirty().
|
2001-07-04 20:15:18 +00:00
|
|
|
*/
|
|
|
|
void
|
2012-06-20 23:25:47 +00:00
|
|
|
vm_page_dirty_KBI(vm_page_t m)
|
2001-07-04 20:15:18 +00:00
|
|
|
{
|
2009-05-30 22:06:58 +00:00
|
|
|
|
2016-11-15 18:22:50 +00:00
|
|
|
/* Refer to this operation by its public name. */
|
2019-10-15 03:45:41 +00:00
|
|
|
KASSERT(vm_page_all_valid(m), ("vm_page_dirty: page is invalid!"));
|
2001-07-04 20:15:18 +00:00
|
|
|
m->dirty = VM_PAGE_BITS_ALL;
|
|
|
|
}
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_page_insert: [ internal use only ]
|
|
|
|
*
|
1998-12-23 01:52:47 +00:00
|
|
|
* Inserts the given mem entry into the object and object list.
|
|
|
|
*
|
2012-10-03 05:06:45 +00:00
|
|
|
* The object must be locked.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2013-08-09 11:28:55 +00:00
|
|
|
int
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_insert(vm_page_t m, vm_object_t object, vm_pindex_t pindex)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2013-05-12 16:50:18 +00:00
|
|
|
vm_page_t mpred;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
2013-05-12 16:50:18 +00:00
|
|
|
mpred = vm_radix_lookup_le(&object->rtree, pindex);
|
2013-08-09 11:28:55 +00:00
|
|
|
return (vm_page_insert_after(m, object, pindex, mpred));
|
2013-05-12 16:50:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_insert_after:
|
|
|
|
*
|
|
|
|
* Inserts the page "m" into the specified object at offset "pindex".
|
|
|
|
*
|
|
|
|
* The page "mpred" must immediately precede the offset "pindex" within
|
|
|
|
* the specified object.
|
|
|
|
*
|
|
|
|
* The object must be locked.
|
|
|
|
*/
|
2013-08-09 11:28:55 +00:00
|
|
|
static int
|
2013-05-12 16:50:18 +00:00
|
|
|
vm_page_insert_after(vm_page_t m, vm_object_t object, vm_pindex_t pindex,
|
|
|
|
vm_page_t mpred)
|
|
|
|
{
|
|
|
|
vm_page_t msucc;
|
|
|
|
|
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
|
|
|
KASSERT(m->object == NULL,
|
|
|
|
("vm_page_insert_after: page already inserted"));
|
|
|
|
if (mpred != NULL) {
|
2013-09-17 07:35:26 +00:00
|
|
|
KASSERT(mpred->object == object,
|
2013-05-12 16:50:18 +00:00
|
|
|
("vm_page_insert_after: object doesn't contain mpred"));
|
|
|
|
KASSERT(mpred->pindex < pindex,
|
|
|
|
("vm_page_insert_after: mpred doesn't precede pindex"));
|
|
|
|
msucc = TAILQ_NEXT(mpred, listq);
|
|
|
|
} else
|
|
|
|
msucc = TAILQ_FIRST(&object->memq);
|
|
|
|
if (msucc != NULL)
|
|
|
|
KASSERT(msucc->pindex > pindex,
|
|
|
|
("vm_page_insert_after: msucc doesn't succeed pindex"));
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
* Record the object/offset pair in this page.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1996-01-19 04:00:31 +00:00
|
|
|
m->object = object;
|
|
|
|
m->pindex = pindex;
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
m->ref_count |= VPRC_OBJREF;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
2002-10-18 17:24:30 +00:00
|
|
|
* Now link into the object's ordered list of backed pages.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2013-08-09 11:28:55 +00:00
|
|
|
if (vm_radix_insert(&object->rtree, m)) {
|
2016-06-02 16:58:47 +00:00
|
|
|
m->object = NULL;
|
|
|
|
m->pindex = 0;
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
m->ref_count &= ~VPRC_OBJREF;
|
2013-08-09 11:28:55 +00:00
|
|
|
return (1);
|
|
|
|
}
|
|
|
|
vm_page_insert_radixdone(m, object, mpred);
|
2022-10-20 12:55:38 +00:00
|
|
|
vm_pager_page_inserted(object, m);
|
2013-08-09 11:28:55 +00:00
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_insert_radixdone:
|
|
|
|
*
|
|
|
|
* Complete page "m" insertion into the specified object after the
|
|
|
|
* radix trie hooking.
|
|
|
|
*
|
|
|
|
* The page "mpred" must precede the offset "m->pindex" within the
|
|
|
|
* specified object.
|
|
|
|
*
|
|
|
|
* The object must be locked.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
vm_page_insert_radixdone(vm_page_t m, vm_object_t object, vm_page_t mpred)
|
|
|
|
{
|
|
|
|
|
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
|
|
|
KASSERT(object != NULL && m->object == object,
|
|
|
|
("vm_page_insert_radixdone: page %p has inconsistent object", m));
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT((m->ref_count & VPRC_OBJREF) != 0,
|
|
|
|
("vm_page_insert_radixdone: page %p is missing object ref", m));
|
2013-08-09 11:28:55 +00:00
|
|
|
if (mpred != NULL) {
|
2013-09-17 07:35:26 +00:00
|
|
|
KASSERT(mpred->object == object,
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
("vm_page_insert_radixdone: object doesn't contain mpred"));
|
2013-08-09 11:28:55 +00:00
|
|
|
KASSERT(mpred->pindex < m->pindex,
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
("vm_page_insert_radixdone: mpred doesn't precede pindex"));
|
2013-08-09 11:28:55 +00:00
|
|
|
}
|
|
|
|
|
2013-05-12 16:50:18 +00:00
|
|
|
if (mpred != NULL)
|
|
|
|
TAILQ_INSERT_AFTER(&object->memq, mpred, m, listq);
|
|
|
|
else
|
|
|
|
TAILQ_INSERT_HEAD(&object->memq, m, listq);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
2012-10-03 05:06:45 +00:00
|
|
|
* Show that the object has one more resident page.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
object->resident_page_count++;
|
2012-10-03 05:06:45 +00:00
|
|
|
|
2005-03-15 14:14:09 +00:00
|
|
|
/*
|
|
|
|
* Hold the vnode until the last page is released.
|
|
|
|
*/
|
|
|
|
if (object->resident_page_count == 1 && object->type == OBJT_VNODE)
|
2012-10-03 05:06:45 +00:00
|
|
|
vhold(object->handle);
|
1999-02-24 21:26:26 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Since we are inserting a new and possibly dirty page,
|
2019-10-29 21:06:34 +00:00
|
|
|
* update the object's generation count.
|
1999-02-24 21:26:26 +00:00
|
|
|
*/
|
2012-06-16 18:56:19 +00:00
|
|
|
if (pmap_page_is_write_mapped(m))
|
2001-10-26 00:08:05 +00:00
|
|
|
vm_object_set_writeable_dirty(object);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
* Do the work to remove a page from its object. The caller is responsible for
|
|
|
|
* updating the page's fields to reflect this removal.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
static void
|
|
|
|
vm_page_object_remove(vm_page_t m)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
1998-02-05 03:32:49 +00:00
|
|
|
vm_object_t object;
|
2022-04-13 23:08:20 +00:00
|
|
|
vm_page_t mrem __diagused;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2019-12-22 06:56:44 +00:00
|
|
|
vm_page_assert_xbusied(m);
|
2019-06-26 17:37:51 +00:00
|
|
|
object = m->object;
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT((m->ref_count & VPRC_OBJREF) != 0,
|
|
|
|
("page %p is missing its object ref", m));
|
2019-10-15 03:38:02 +00:00
|
|
|
|
2019-12-15 03:15:06 +00:00
|
|
|
/* Deferred free of swap space. */
|
|
|
|
if ((m->a.flags & PGA_SWAP_FREE) != 0)
|
|
|
|
vm_pager_page_unswapped(m);
|
|
|
|
|
2022-10-20 12:55:38 +00:00
|
|
|
vm_pager_page_removed(object, m);
|
|
|
|
|
2020-02-27 02:37:27 +00:00
|
|
|
m->object = NULL;
|
2016-12-08 04:29:29 +00:00
|
|
|
mrem = vm_radix_remove(&object->rtree, m->pindex);
|
|
|
|
KASSERT(mrem == m, ("removed page %p, expected page %p", mrem, m));
|
1999-01-21 08:29:12 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Now remove from the object's list of backed pages.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1998-02-05 03:32:49 +00:00
|
|
|
TAILQ_REMOVE(&object->memq, m, listq);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* And show that the object has one fewer resident page.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1998-02-05 03:32:49 +00:00
|
|
|
object->resident_page_count--;
|
2012-10-03 05:06:45 +00:00
|
|
|
|
2005-03-15 14:14:09 +00:00
|
|
|
/*
|
|
|
|
* The vnode may now be recycled.
|
|
|
|
*/
|
|
|
|
if (object->resident_page_count == 0 && object->type == OBJT_VNODE)
|
2012-10-03 05:06:45 +00:00
|
|
|
vdrop(object->handle);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
/*
|
|
|
|
* vm_page_remove:
|
|
|
|
*
|
|
|
|
* Removes the specified page from its containing object, but does not
|
|
|
|
* invalidate any backing storage. Returns true if the object's reference
|
|
|
|
* was the last reference to the page, and false otherwise.
|
|
|
|
*
|
2019-12-22 06:56:44 +00:00
|
|
|
* The object must be locked and the page must be exclusively busied.
|
|
|
|
* The exclusive busy will be released on return. If this is not the
|
|
|
|
* final ref and the caller does not hold a wire reference it may not
|
|
|
|
* continue to access the page.
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
*/
|
|
|
|
bool
|
|
|
|
vm_page_remove(vm_page_t m)
|
2019-12-22 06:56:44 +00:00
|
|
|
{
|
|
|
|
bool dropped;
|
|
|
|
|
|
|
|
dropped = vm_page_remove_xbusy(m);
|
|
|
|
vm_page_xunbusy(m);
|
|
|
|
|
|
|
|
return (dropped);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_remove_xbusy
|
|
|
|
*
|
|
|
|
* Removes the page but leaves the xbusy held. Returns true if this
|
|
|
|
* removed the final ref and false otherwise.
|
|
|
|
*/
|
|
|
|
bool
|
|
|
|
vm_page_remove_xbusy(vm_page_t m)
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
vm_page_object_remove(m);
|
|
|
|
return (vm_page_drop(m, VPRC_OBJREF) == VPRC_OBJREF);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_lookup:
|
|
|
|
*
|
|
|
|
* Returns the page associated with the object/offset
|
|
|
|
* pair specified; if none is found, NULL is returned.
|
|
|
|
*
|
2002-10-18 17:24:30 +00:00
|
|
|
* The object must be locked.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
vm_page_t
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_lookup(vm_object_t object, vm_pindex_t pindex)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
|
|
|
|
2013-05-17 18:49:43 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(object);
|
Sync back vmcontention branch into HEAD:
Replace the per-object resident and cached pages splay tree with a
path-compressed multi-digit radix trie.
Along with this, switch also the x86-specific handling of idle page
tables to using the radix trie.
This change is supposed to do the following:
- Allowing the acquisition of read locking for lookup operations of the
resident/cached pages collections as the per-vm_page_t splay iterators
are now removed.
- Increase the scalability of the operations on the page collections.
The radix trie does rely on the consumers locking to ensure atomicity of
its operations. In order to avoid deadlocks the bisection nodes are
pre-allocated in the UMA zone. This can be done safely because the
algorithm needs at maximum one new node per insert which means the
maximum number of the desired nodes is the number of available physical
frames themselves. However, not all the times a new bisection node is
really needed.
The radix trie implements path-compression because UFS indirect blocks
can lead to several objects with a very sparse trie, increasing the number
of levels to usually scan. It also helps in the nodes pre-fetching by
introducing the single node per-insert property.
This code is not generalized (yet) because of the possible loss of
performance by having much of the sizes in play configurable.
However, efforts to make this code more general and then reusable in
further different consumers might be really done.
The only KPI change is the removal of the function vm_page_splay() which
is now reaped.
The only KBI change, instead, is the removal of the left/right iterators
from struct vm_page, which are now reaped.
Further technical notes broken into mealpieces can be retrieved from the
svn branch:
http://svn.freebsd.org/base/user/attilio/vmcontention/
Sponsored by: EMC / Isilon storage division
In collaboration with: alc, jeff
Tested by: flo, pho, jhb, davide
Tested by: ian (arm)
Tested by: andreast (powerpc)
2013-03-18 00:25:02 +00:00
|
|
|
return (vm_radix_lookup(&object->rtree, pindex));
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2020-10-09 23:49:42 +00:00
|
|
|
/*
|
|
|
|
* vm_page_lookup_unlocked:
|
|
|
|
*
|
|
|
|
* Returns the page associated with the object/offset pair specified;
|
|
|
|
* if none is found, NULL is returned. The page may be no longer be
|
|
|
|
* present in the object at the time that this function returns. Only
|
|
|
|
* useful for opportunistic checks such as inmem().
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_lookup_unlocked(vm_object_t object, vm_pindex_t pindex)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (vm_radix_lookup_unlocked(&object->rtree, pindex));
|
|
|
|
}
|
|
|
|
|
2020-02-28 21:42:48 +00:00
|
|
|
/*
|
|
|
|
* vm_page_relookup:
|
|
|
|
*
|
|
|
|
* Returns a page that must already have been busied by
|
|
|
|
* the caller. Used for bogus page replacement.
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_relookup(vm_object_t object, vm_pindex_t pindex)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
|
|
|
m = vm_radix_lookup_unlocked(&object->rtree, pindex);
|
2020-03-11 22:25:45 +00:00
|
|
|
KASSERT(m != NULL && (vm_page_busied(m) || vm_page_wired(m)) &&
|
2020-02-28 21:42:48 +00:00
|
|
|
m->object == object && m->pindex == pindex,
|
|
|
|
("vm_page_relookup: Invalid page %p", m));
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
2020-02-27 02:37:27 +00:00
|
|
|
/*
|
|
|
|
* This should only be used by lockless functions for releasing transient
|
|
|
|
* incorrect acquires. The page may have been freed after we acquired a
|
|
|
|
* busy lock. In this case busy_lock == VPB_FREED and we have nothing
|
|
|
|
* further to do.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
vm_page_busy_release(vm_page_t m)
|
|
|
|
{
|
|
|
|
u_int x;
|
|
|
|
|
2020-07-29 19:38:49 +00:00
|
|
|
x = vm_page_busy_fetch(m);
|
2020-02-27 02:37:27 +00:00
|
|
|
for (;;) {
|
|
|
|
if (x == VPB_FREED)
|
|
|
|
break;
|
|
|
|
if ((x & VPB_BIT_SHARED) != 0 && VPB_SHARERS(x) > 1) {
|
|
|
|
if (atomic_fcmpset_int(&m->busy_lock, &x,
|
|
|
|
x - VPB_ONE_SHARER))
|
|
|
|
break;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
KASSERT((x & VPB_BIT_SHARED) != 0 ||
|
|
|
|
(x & ~VPB_BIT_WAITERS) == VPB_CURTHREAD_EXCLUSIVE,
|
|
|
|
("vm_page_busy_release: %p xbusy not owned.", m));
|
|
|
|
if (!atomic_fcmpset_rel_int(&m->busy_lock, &x, VPB_UNBUSIED))
|
|
|
|
continue;
|
|
|
|
if ((x & VPB_BIT_WAITERS) != 0)
|
|
|
|
wakeup(m);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2010-07-04 11:13:33 +00:00
|
|
|
/*
|
|
|
|
* vm_page_find_least:
|
|
|
|
*
|
|
|
|
* Returns the page associated with the object with least pindex
|
|
|
|
* greater than or equal to the parameter pindex, or NULL.
|
|
|
|
*
|
|
|
|
* The object must be locked.
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_find_least(vm_object_t object, vm_pindex_t pindex)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
2013-05-21 20:38:19 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(object);
|
Sync back vmcontention branch into HEAD:
Replace the per-object resident and cached pages splay tree with a
path-compressed multi-digit radix trie.
Along with this, switch also the x86-specific handling of idle page
tables to using the radix trie.
This change is supposed to do the following:
- Allowing the acquisition of read locking for lookup operations of the
resident/cached pages collections as the per-vm_page_t splay iterators
are now removed.
- Increase the scalability of the operations on the page collections.
The radix trie does rely on the consumers locking to ensure atomicity of
its operations. In order to avoid deadlocks the bisection nodes are
pre-allocated in the UMA zone. This can be done safely because the
algorithm needs at maximum one new node per insert which means the
maximum number of the desired nodes is the number of available physical
frames themselves. However, not all the times a new bisection node is
really needed.
The radix trie implements path-compression because UFS indirect blocks
can lead to several objects with a very sparse trie, increasing the number
of levels to usually scan. It also helps in the nodes pre-fetching by
introducing the single node per-insert property.
This code is not generalized (yet) because of the possible loss of
performance by having much of the sizes in play configurable.
However, efforts to make this code more general and then reusable in
further different consumers might be really done.
The only KPI change is the removal of the function vm_page_splay() which
is now reaped.
The only KBI change, instead, is the removal of the left/right iterators
from struct vm_page, which are now reaped.
Further technical notes broken into mealpieces can be retrieved from the
svn branch:
http://svn.freebsd.org/base/user/attilio/vmcontention/
Sponsored by: EMC / Isilon storage division
In collaboration with: alc, jeff
Tested by: flo, pho, jhb, davide
Tested by: ian (arm)
Tested by: andreast (powerpc)
2013-03-18 00:25:02 +00:00
|
|
|
if ((m = TAILQ_FIRST(&object->memq)) != NULL && m->pindex < pindex)
|
|
|
|
m = vm_radix_lookup_ge(&object->rtree, pindex);
|
2010-07-04 11:13:33 +00:00
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
2010-06-21 23:27:24 +00:00
|
|
|
/*
|
|
|
|
* Returns the given page's successor (by pindex) within the object if it is
|
|
|
|
* resident; if none is found, NULL is returned.
|
|
|
|
*
|
|
|
|
* The object must be locked.
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_next(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_page_t next;
|
|
|
|
|
2016-04-29 17:35:28 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(m->object);
|
2016-12-30 17:37:06 +00:00
|
|
|
if ((next = TAILQ_NEXT(m, listq)) != NULL) {
|
|
|
|
MPASS(next->object == m->object);
|
|
|
|
if (next->pindex != m->pindex + 1)
|
|
|
|
next = NULL;
|
|
|
|
}
|
2010-06-21 23:27:24 +00:00
|
|
|
return (next);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Returns the given page's predecessor (by pindex) within the object if it is
|
|
|
|
* resident; if none is found, NULL is returned.
|
|
|
|
*
|
|
|
|
* The object must be locked.
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_prev(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_page_t prev;
|
|
|
|
|
2016-04-29 17:35:28 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(m->object);
|
2016-12-30 17:37:06 +00:00
|
|
|
if ((prev = TAILQ_PREV(m, pglist, listq)) != NULL) {
|
|
|
|
MPASS(prev->object == m->object);
|
|
|
|
if (prev->pindex != m->pindex - 1)
|
|
|
|
prev = NULL;
|
|
|
|
}
|
2010-06-21 23:27:24 +00:00
|
|
|
return (prev);
|
|
|
|
}
|
|
|
|
|
2013-08-09 11:28:55 +00:00
|
|
|
/*
|
|
|
|
* Uses the page mnew as a replacement for an existing page at index
|
|
|
|
* pindex which must be already present in the object.
|
2019-12-22 06:56:44 +00:00
|
|
|
*
|
|
|
|
* Both pages must be exclusively busied on enter. The old page is
|
|
|
|
* unbusied on exit.
|
|
|
|
*
|
|
|
|
* A return value of true means mold is now free. If this is not the
|
|
|
|
* final ref and the caller does not hold a wire reference it may not
|
|
|
|
* continue to access the page.
|
2013-08-09 11:28:55 +00:00
|
|
|
*/
|
2019-12-22 06:56:44 +00:00
|
|
|
static bool
|
|
|
|
vm_page_replace_hold(vm_page_t mnew, vm_object_t object, vm_pindex_t pindex,
|
|
|
|
vm_page_t mold)
|
2013-08-09 11:28:55 +00:00
|
|
|
{
|
2022-04-13 23:08:20 +00:00
|
|
|
vm_page_t mret __diagused;
|
2019-12-22 06:56:44 +00:00
|
|
|
bool dropped;
|
2013-08-09 11:28:55 +00:00
|
|
|
|
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
2019-12-22 06:56:44 +00:00
|
|
|
vm_page_assert_xbusied(mold);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT(mnew->object == NULL && (mnew->ref_count & VPRC_OBJREF) == 0,
|
2018-03-18 16:35:40 +00:00
|
|
|
("vm_page_replace: page %p already in object", mnew));
|
2013-08-09 11:28:55 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* This function mostly follows vm_page_insert() and
|
|
|
|
* vm_page_remove() without the radix, object count and vnode
|
|
|
|
* dance. Double check such functions for more comments.
|
|
|
|
*/
|
|
|
|
|
|
|
|
mnew->object = object;
|
|
|
|
mnew->pindex = pindex;
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
atomic_set_int(&mnew->ref_count, VPRC_OBJREF);
|
2019-12-22 06:56:44 +00:00
|
|
|
mret = vm_radix_replace(&object->rtree, mnew);
|
|
|
|
KASSERT(mret == mold,
|
|
|
|
("invalid page replacement, mold=%p, mret=%p", mold, mret));
|
2019-12-21 19:04:05 +00:00
|
|
|
KASSERT((mold->oflags & VPO_UNMANAGED) ==
|
|
|
|
(mnew->oflags & VPO_UNMANAGED),
|
|
|
|
("vm_page_replace: mismatched VPO_UNMANAGED"));
|
2013-08-09 11:28:55 +00:00
|
|
|
|
2015-12-10 22:57:27 +00:00
|
|
|
/* Keep the resident page list in sorted order. */
|
|
|
|
TAILQ_INSERT_AFTER(&object->memq, mold, mnew, listq);
|
2013-08-09 11:28:55 +00:00
|
|
|
TAILQ_REMOVE(&object->memq, mold, listq);
|
|
|
|
mold->object = NULL;
|
|
|
|
|
2015-12-10 22:57:27 +00:00
|
|
|
/*
|
|
|
|
* The object's resident_page_count does not change because we have
|
2019-10-29 21:06:34 +00:00
|
|
|
* swapped one page for another, but the generation count should
|
|
|
|
* change if the page is dirty.
|
2015-12-10 22:57:27 +00:00
|
|
|
*/
|
2013-08-09 11:28:55 +00:00
|
|
|
if (pmap_page_is_write_mapped(mnew))
|
|
|
|
vm_object_set_writeable_dirty(object);
|
2019-12-22 06:56:44 +00:00
|
|
|
dropped = vm_page_drop(mold, VPRC_OBJREF) == VPRC_OBJREF;
|
|
|
|
vm_page_xunbusy(mold);
|
|
|
|
|
|
|
|
return (dropped);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_replace(vm_page_t mnew, vm_object_t object, vm_pindex_t pindex,
|
|
|
|
vm_page_t mold)
|
|
|
|
{
|
|
|
|
|
2019-12-27 01:50:16 +00:00
|
|
|
vm_page_assert_xbusied(mnew);
|
|
|
|
|
2019-12-22 06:56:44 +00:00
|
|
|
if (vm_page_replace_hold(mnew, object, pindex, mold))
|
|
|
|
vm_page_free(mold);
|
2013-08-09 11:28:55 +00:00
|
|
|
}
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_page_rename:
|
|
|
|
*
|
|
|
|
* Move the given memory entry from its
|
|
|
|
* current object to the specified target object/offset.
|
|
|
|
*
|
1999-01-21 08:29:12 +00:00
|
|
|
* Note: swap associated with the page must be invalidated by the move. We
|
|
|
|
* have to do this for several reasons: (1) we aren't freeing the
|
|
|
|
* page, (2) we are dirtying the page, (3) the VM system is probably
|
|
|
|
* moving the page from object A to B, and will then later move
|
|
|
|
* the backing store from A to B and we can't have a conflict.
|
|
|
|
*
|
|
|
|
* Note: we *always* dirty the page. It is necessary both for the
|
|
|
|
* fact that we moved it, and because we may be invalidating
|
2016-12-12 17:47:09 +00:00
|
|
|
* swap.
|
2012-10-03 05:06:45 +00:00
|
|
|
*
|
2013-08-09 11:28:55 +00:00
|
|
|
* The objects must be locked.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2013-08-09 11:28:55 +00:00
|
|
|
int
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_rename(vm_page_t m, vm_object_t new_object, vm_pindex_t new_pindex)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2013-08-09 11:28:55 +00:00
|
|
|
vm_page_t mpred;
|
|
|
|
vm_pindex_t opidx;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2013-08-09 11:28:55 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(new_object);
|
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT(m->ref_count != 0, ("vm_page_rename: page %p has no refs", m));
|
2013-08-09 11:28:55 +00:00
|
|
|
mpred = vm_radix_lookup_le(&new_object->rtree, new_pindex);
|
|
|
|
KASSERT(mpred == NULL || mpred->pindex != new_pindex,
|
|
|
|
("vm_page_rename: pindex already renamed"));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a custom version of vm_page_insert() which does not depend
|
|
|
|
* by m_prev and can cheat on the implementation aspects of the
|
|
|
|
* function.
|
|
|
|
*/
|
|
|
|
opidx = m->pindex;
|
|
|
|
m->pindex = new_pindex;
|
|
|
|
if (vm_radix_insert(&new_object->rtree, m)) {
|
|
|
|
m->pindex = opidx;
|
|
|
|
return (1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The operation cannot fail anymore. The removal must happen before
|
|
|
|
* the listq iterator is tainted.
|
|
|
|
*/
|
|
|
|
m->pindex = opidx;
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
vm_page_object_remove(m);
|
2013-08-09 11:28:55 +00:00
|
|
|
|
|
|
|
/* Return back to the new pindex to complete vm_page_insert(). */
|
|
|
|
m->pindex = new_pindex;
|
|
|
|
m->object = new_object;
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
|
2013-08-09 11:28:55 +00:00
|
|
|
vm_page_insert_radixdone(m, new_object, mpred);
|
1999-01-24 06:00:31 +00:00
|
|
|
vm_page_dirty(m);
|
2022-10-20 12:55:38 +00:00
|
|
|
vm_pager_page_inserted(new_object, m);
|
2013-08-09 11:28:55 +00:00
|
|
|
return (0);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_alloc:
|
|
|
|
*
|
2011-10-27 17:29:19 +00:00
|
|
|
* Allocate and return a page that is associated with the specified
|
2013-08-09 11:11:11 +00:00
|
|
|
* object and offset pair. By default, this page is exclusive busied.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
2010-07-03 18:25:37 +00:00
|
|
|
* The caller must always specify an allocation class.
|
|
|
|
*
|
|
|
|
* allocation classes:
|
1995-03-01 23:30:04 +00:00
|
|
|
* VM_ALLOC_NORMAL normal process request
|
|
|
|
* VM_ALLOC_SYSTEM system *really* needs a page
|
|
|
|
* VM_ALLOC_INTERRUPT interrupt time request
|
2010-07-03 18:25:37 +00:00
|
|
|
*
|
|
|
|
* optional allocation flags:
|
2011-11-06 02:03:27 +00:00
|
|
|
* VM_ALLOC_COUNT(number) the number of additional pages that the caller
|
|
|
|
* intends to allocate
|
2013-08-09 11:11:11 +00:00
|
|
|
* VM_ALLOC_NOBUSY do not exclusive busy the page
|
2012-01-27 20:18:31 +00:00
|
|
|
* VM_ALLOC_NODUMP do not include the page in a kernel core dump
|
2013-08-09 11:11:11 +00:00
|
|
|
* VM_ALLOC_SBUSY shared busy the allocated page
|
2011-10-27 17:29:19 +00:00
|
|
|
* VM_ALLOC_WIRED wire the allocated page
|
|
|
|
* VM_ALLOC_ZERO prefer a zeroed page
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1994-05-25 09:21:21 +00:00
|
|
|
vm_page_t
|
2002-07-18 04:08:10 +00:00
|
|
|
vm_page_alloc(vm_object_t object, vm_pindex_t pindex, int req)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2017-08-15 16:39:49 +00:00
|
|
|
|
2021-10-20 00:25:14 +00:00
|
|
|
return (vm_page_alloc_after(object, pindex, req,
|
|
|
|
vm_radix_lookup_le(&object->rtree, pindex)));
|
2017-08-15 16:39:49 +00:00
|
|
|
}
|
|
|
|
|
2017-11-28 23:18:35 +00:00
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_domain(vm_object_t object, vm_pindex_t pindex, int domain,
|
|
|
|
int req)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (vm_page_alloc_domain_after(object, pindex, domain, req,
|
2021-10-20 00:25:14 +00:00
|
|
|
vm_radix_lookup_le(&object->rtree, pindex)));
|
2017-11-28 23:18:35 +00:00
|
|
|
}
|
|
|
|
|
2017-08-15 16:39:49 +00:00
|
|
|
/*
|
|
|
|
* Allocate a page in the specified object with the given page index. To
|
|
|
|
* optimize insertion of the page into the object, the caller must also specifiy
|
|
|
|
* the resident page in the object with largest index smaller than the given
|
|
|
|
* page index, or NULL if no such page exists.
|
|
|
|
*/
|
|
|
|
vm_page_t
|
2017-11-28 23:18:35 +00:00
|
|
|
vm_page_alloc_after(vm_object_t object, vm_pindex_t pindex,
|
|
|
|
int req, vm_page_t mpred)
|
|
|
|
{
|
2018-01-12 22:48:23 +00:00
|
|
|
struct vm_domainset_iter di;
|
2017-11-28 23:18:35 +00:00
|
|
|
vm_page_t m;
|
2018-01-12 22:48:23 +00:00
|
|
|
int domain;
|
2017-11-28 23:18:35 +00:00
|
|
|
|
2018-03-29 02:54:50 +00:00
|
|
|
vm_domainset_iter_page_init(&di, object, pindex, &domain, &req);
|
2018-01-12 22:48:23 +00:00
|
|
|
do {
|
2017-11-28 23:18:35 +00:00
|
|
|
m = vm_page_alloc_domain_after(object, pindex, domain, req,
|
|
|
|
mpred);
|
|
|
|
if (m != NULL)
|
|
|
|
break;
|
2018-10-23 16:35:58 +00:00
|
|
|
} while (vm_domainset_iter_page(&di, object, &domain) == 0);
|
2017-11-28 23:18:35 +00:00
|
|
|
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
/*
|
|
|
|
* Returns true if the number of free pages exceeds the minimum
|
|
|
|
* for the request class and false otherwise.
|
|
|
|
*/
|
2019-11-22 16:31:10 +00:00
|
|
|
static int
|
|
|
|
_vm_domain_allocate(struct vm_domain *vmd, int req_class, int npages)
|
2018-02-06 22:10:07 +00:00
|
|
|
{
|
2018-03-22 19:21:11 +00:00
|
|
|
u_int limit, old, new;
|
2018-02-06 22:10:07 +00:00
|
|
|
|
2019-11-22 16:31:10 +00:00
|
|
|
if (req_class == VM_ALLOC_INTERRUPT)
|
2018-03-22 19:21:11 +00:00
|
|
|
limit = 0;
|
2019-11-22 16:31:10 +00:00
|
|
|
else if (req_class == VM_ALLOC_SYSTEM)
|
2018-03-22 19:21:11 +00:00
|
|
|
limit = vmd->vmd_interrupt_free_min;
|
|
|
|
else
|
|
|
|
limit = vmd->vmd_free_reserved;
|
2018-02-06 22:10:07 +00:00
|
|
|
|
2018-03-22 19:21:11 +00:00
|
|
|
/*
|
|
|
|
* Attempt to reserve the pages. Fail if we're below the limit.
|
|
|
|
*/
|
|
|
|
limit += npages;
|
|
|
|
old = vmd->vmd_free_count;
|
|
|
|
do {
|
|
|
|
if (old < limit)
|
|
|
|
return (0);
|
|
|
|
new = old - npages;
|
|
|
|
} while (atomic_fcmpset_int(&vmd->vmd_free_count, &old, new) == 0);
|
2018-02-06 22:10:07 +00:00
|
|
|
|
2018-03-22 19:21:11 +00:00
|
|
|
/* Wake the page daemon if we've crossed the threshold. */
|
|
|
|
if (vm_paging_needed(vmd, new) && !vm_paging_needed(vmd, old))
|
|
|
|
pagedaemon_wakeup(vmd->vmd_domain);
|
|
|
|
|
|
|
|
/* Only update bitsets on transitions. */
|
|
|
|
if ((old >= vmd->vmd_free_min && new < vmd->vmd_free_min) ||
|
|
|
|
(old >= vmd->vmd_free_severe && new < vmd->vmd_free_severe))
|
|
|
|
vm_domain_set(vmd);
|
|
|
|
|
|
|
|
return (1);
|
2018-02-06 22:10:07 +00:00
|
|
|
}
|
|
|
|
|
2019-11-22 16:31:10 +00:00
|
|
|
int
|
|
|
|
vm_domain_allocate(struct vm_domain *vmd, int req, int npages)
|
|
|
|
{
|
|
|
|
int req_class;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The page daemon is allowed to dig deeper into the free page list.
|
|
|
|
*/
|
|
|
|
req_class = req & VM_ALLOC_CLASS_MASK;
|
|
|
|
if (curproc == pageproc && req_class != VM_ALLOC_INTERRUPT)
|
|
|
|
req_class = VM_ALLOC_SYSTEM;
|
|
|
|
return (_vm_domain_allocate(vmd, req_class, npages));
|
|
|
|
}
|
|
|
|
|
2017-11-28 23:18:35 +00:00
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_domain_after(vm_object_t object, vm_pindex_t pindex, int domain,
|
|
|
|
int req, vm_page_t mpred)
|
2017-08-15 16:39:49 +00:00
|
|
|
{
|
2018-02-06 22:10:07 +00:00
|
|
|
struct vm_domain *vmd;
|
2017-08-15 16:39:49 +00:00
|
|
|
vm_page_t m;
|
2021-10-20 00:25:14 +00:00
|
|
|
int flags;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2021-10-20 00:25:14 +00:00
|
|
|
#define VPA_FLAGS (VM_ALLOC_CLASS_MASK | VM_ALLOC_WAITFAIL | \
|
|
|
|
VM_ALLOC_NOWAIT | VM_ALLOC_NOBUSY | \
|
|
|
|
VM_ALLOC_SBUSY | VM_ALLOC_WIRED | \
|
|
|
|
VM_ALLOC_NODUMP | VM_ALLOC_ZERO | VM_ALLOC_COUNT_MASK)
|
|
|
|
KASSERT((req & ~VPA_FLAGS) == 0,
|
|
|
|
("invalid request %#x", req));
|
|
|
|
KASSERT(((req & (VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)) !=
|
2013-08-09 11:11:11 +00:00
|
|
|
(VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)),
|
2021-10-20 00:25:14 +00:00
|
|
|
("invalid request %#x", req));
|
2017-08-15 16:39:49 +00:00
|
|
|
KASSERT(mpred == NULL || mpred->pindex < pindex,
|
|
|
|
("mpred %p doesn't precede pindex 0x%jx", mpred,
|
|
|
|
(uintmax_t)pindex));
|
2021-10-20 00:25:14 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
2002-11-01 00:59:03 +00:00
|
|
|
|
2019-07-02 19:51:40 +00:00
|
|
|
flags = 0;
|
2017-11-28 23:18:35 +00:00
|
|
|
m = NULL;
|
2022-10-20 13:15:46 +00:00
|
|
|
if (!vm_pager_can_alloc_page(object, pindex))
|
|
|
|
return (NULL);
|
2019-07-02 19:51:40 +00:00
|
|
|
again:
|
2018-02-06 22:10:07 +00:00
|
|
|
#if VM_NRESERVLEVEL > 0
|
2018-03-22 19:21:11 +00:00
|
|
|
/*
|
|
|
|
* Can we allocate the page from a reservation?
|
|
|
|
*/
|
2018-02-06 22:10:07 +00:00
|
|
|
if (vm_object_reserv(object) &&
|
2019-06-03 05:15:36 +00:00
|
|
|
(m = vm_reserv_alloc_page(object, pindex, domain, req, mpred)) !=
|
|
|
|
NULL) {
|
2018-02-06 22:10:07 +00:00
|
|
|
goto found;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
vmd = VM_DOMAIN(domain);
|
2021-10-20 00:25:14 +00:00
|
|
|
if (vmd->vmd_pgcache[VM_FREEPOOL_DEFAULT].zone != NULL) {
|
|
|
|
m = uma_zalloc(vmd->vmd_pgcache[VM_FREEPOOL_DEFAULT].zone,
|
|
|
|
M_NOWAIT | M_NOVM);
|
2019-07-02 19:51:40 +00:00
|
|
|
if (m != NULL) {
|
|
|
|
flags |= PG_PCPU_CACHE;
|
2018-04-01 04:50:05 +00:00
|
|
|
goto found;
|
2019-07-02 19:51:40 +00:00
|
|
|
}
|
2018-04-01 04:50:05 +00:00
|
|
|
}
|
2018-03-22 19:21:11 +00:00
|
|
|
if (vm_domain_allocate(vmd, req, 1)) {
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
2018-03-22 19:21:11 +00:00
|
|
|
* If not, allocate it from the free page queues.
|
1999-02-15 06:52:14 +00:00
|
|
|
*/
|
2018-03-22 19:21:11 +00:00
|
|
|
vm_domain_free_lock(vmd);
|
2021-10-20 00:25:14 +00:00
|
|
|
m = vm_phys_alloc_pages(domain, VM_FREEPOOL_DEFAULT, 0);
|
2018-03-22 19:21:11 +00:00
|
|
|
vm_domain_free_unlock(vmd);
|
|
|
|
if (m == NULL) {
|
|
|
|
vm_domain_freecnt_inc(vmd, 1);
|
2007-12-29 19:53:04 +00:00
|
|
|
#if VM_NRESERVLEVEL > 0
|
2018-03-22 19:21:11 +00:00
|
|
|
if (vm_reserv_reclaim_inactive(domain))
|
|
|
|
goto again;
|
2007-12-29 19:53:04 +00:00
|
|
|
#endif
|
|
|
|
}
|
2017-11-28 23:18:35 +00:00
|
|
|
}
|
|
|
|
if (m == NULL) {
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
Change the management of cached pages (PQ_CACHE) in two fundamental
ways:
(1) Cached pages are no longer kept in the object's resident page
splay tree and memq. Instead, they are kept in a separate per-object
splay tree of cached pages. However, access to this new per-object
splay tree is synchronized by the _free_ page queues lock, not to be
confused with the heavily contended page queues lock. Consequently, a
cached page can be reclaimed by vm_page_alloc(9) without acquiring the
object's lock or the page queues lock.
This solves a problem independently reported by tegge@ and Isilon.
Specifically, they observed the page daemon consuming a great deal of
CPU time because of pages bouncing back and forth between the cache
queue (PQ_CACHE) and the inactive queue (PQ_INACTIVE). The source of
this problem turned out to be a deadlock avoidance strategy employed
when selecting a cached page to reclaim in vm_page_select_cache().
However, the root cause was really that reclaiming a cached page
required the acquisition of an object lock while the page queues lock
was already held. Thus, this change addresses the problem at its
root, by eliminating the need to acquire the object's lock.
Moreover, keeping cached pages in the object's primary splay tree and
memq was, in effect, optimizing for the uncommon case. Cached pages
are reclaimed far, far more often than they are reactivated. Instead,
this change makes reclamation cheaper, especially in terms of
synchronization overhead, and reactivation more expensive, because
reactivated pages will have to be reentered into the object's primary
splay tree and memq.
(2) Cached pages are now stored alongside free pages in the physical
memory allocator's buddy queues, increasing the likelihood that large
allocations of contiguous physical memory (i.e., superpages) will
succeed.
Finally, as a result of this change long-standing restrictions on when
and where a cached page can be reclaimed and returned by
vm_page_alloc(9) are eliminated. Specifically, calls to
vm_page_alloc(9) specifying VM_ALLOC_INTERRUPT can now reclaim and
return a formerly cached page. Consequently, a call to malloc(9)
specifying M_NOWAIT is less likely to fail.
Discussed with: many over the course of the summer, including jeff@,
Justin Husted @ Isilon, peter@, tegge@
Tested by: an earlier version by kris@
Approved by: re (kensmith)
2007-09-25 06:25:06 +00:00
|
|
|
* Not allocatable, give up.
|
1999-02-15 06:52:14 +00:00
|
|
|
*/
|
2018-02-06 22:10:07 +00:00
|
|
|
if (vm_domain_alloc_fail(vmd, object, req))
|
2017-11-08 02:39:37 +00:00
|
|
|
goto again;
|
1999-02-15 06:52:14 +00:00
|
|
|
return (NULL);
|
1995-03-01 23:30:04 +00:00
|
|
|
}
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
2019-07-02 19:51:40 +00:00
|
|
|
* At this point we had better have found a good page.
|
1999-02-15 06:52:14 +00:00
|
|
|
*/
|
2018-02-06 22:10:07 +00:00
|
|
|
found:
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_page_dequeue(m);
|
2016-11-15 18:22:50 +00:00
|
|
|
vm_page_alloc_check(m);
|
1999-01-21 08:29:12 +00:00
|
|
|
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
2013-12-31 18:25:15 +00:00
|
|
|
* Initialize the page. Only the PG_ZERO flag is inherited.
|
1999-02-15 06:52:14 +00:00
|
|
|
*/
|
2021-10-20 00:25:14 +00:00
|
|
|
flags |= m->flags & PG_ZERO;
|
2013-12-31 18:25:15 +00:00
|
|
|
if ((req & VM_ALLOC_NODUMP) != 0)
|
2012-11-21 06:26:18 +00:00
|
|
|
flags |= PG_NODUMP;
|
2003-01-12 23:32:46 +00:00
|
|
|
m->flags = flags;
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.flags = 0;
|
2021-10-20 00:25:14 +00:00
|
|
|
m->oflags = (object->flags & OBJ_UNMANAGED) != 0 ? VPO_UNMANAGED : 0;
|
|
|
|
if ((req & (VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)) == 0)
|
2019-11-24 19:12:23 +00:00
|
|
|
m->busy_lock = VPB_CURTHREAD_EXCLUSIVE;
|
2020-02-04 20:33:01 +00:00
|
|
|
else if ((req & VM_ALLOC_SBUSY) != 0)
|
2013-08-09 11:11:11 +00:00
|
|
|
m->busy_lock = VPB_SHARERS_WORD(1);
|
2020-02-04 20:33:01 +00:00
|
|
|
else
|
|
|
|
m->busy_lock = VPB_UNBUSIED;
|
2002-07-18 04:08:10 +00:00
|
|
|
if (req & VM_ALLOC_WIRED) {
|
2018-02-12 22:53:00 +00:00
|
|
|
vm_wire_add(1);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
m->ref_count = 1;
|
2009-06-21 00:21:33 +00:00
|
|
|
}
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.act_count = 0;
|
1995-01-10 09:19:52 +00:00
|
|
|
|
2021-10-20 00:25:14 +00:00
|
|
|
if (vm_page_insert_after(m, object, pindex, mpred)) {
|
|
|
|
if (req & VM_ALLOC_WIRED) {
|
|
|
|
vm_wire_sub(1);
|
|
|
|
m->ref_count = 0;
|
|
|
|
}
|
|
|
|
KASSERT(m->object == NULL, ("page %p has object", m));
|
|
|
|
m->oflags = VPO_UNMANAGED;
|
|
|
|
m->busy_lock = VPB_UNBUSIED;
|
|
|
|
/* Don't change PG_ZERO. */
|
|
|
|
vm_page_free_toq(m);
|
|
|
|
if (req & VM_ALLOC_WAITFAIL) {
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
vm_radix_wait();
|
|
|
|
VM_OBJECT_WLOCK(object);
|
2013-08-09 11:28:55 +00:00
|
|
|
}
|
2021-10-20 00:25:14 +00:00
|
|
|
return (NULL);
|
|
|
|
}
|
2013-08-09 11:28:55 +00:00
|
|
|
|
2021-10-20 00:25:14 +00:00
|
|
|
/* Ignore device objects; the pager sets "memattr" for them. */
|
|
|
|
if (object->memattr != VM_MEMATTR_DEFAULT &&
|
|
|
|
(object->flags & OBJ_FICTITIOUS) == 0)
|
|
|
|
pmap_page_set_memattr(m, object->memattr);
|
1995-01-10 09:19:52 +00:00
|
|
|
|
1996-01-19 04:00:31 +00:00
|
|
|
return (m);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
/*
|
|
|
|
* vm_page_alloc_contig:
|
|
|
|
*
|
|
|
|
* Allocate a contiguous set of physical pages of the given size "npages"
|
|
|
|
* from the free lists. All of the physical pages must be at or above
|
|
|
|
* the given physical address "low" and below the given physical address
|
|
|
|
* "high". The given value "alignment" determines the alignment of the
|
|
|
|
* first physical page in the set. If the given value "boundary" is
|
|
|
|
* non-zero, then the set of physical pages cannot cross any physical
|
|
|
|
* address boundary that is a multiple of that value. Both "alignment"
|
|
|
|
* and "boundary" must be a power of two.
|
|
|
|
*
|
|
|
|
* If the specified memory attribute, "memattr", is VM_MEMATTR_DEFAULT,
|
|
|
|
* then the memory attribute setting for the physical pages is configured
|
|
|
|
* to the object's memory attribute setting. Otherwise, the memory
|
|
|
|
* attribute setting for the physical pages is configured to "memattr",
|
|
|
|
* overriding the object's memory attribute setting. However, if the
|
|
|
|
* object's memory attribute setting is not VM_MEMATTR_DEFAULT, then the
|
|
|
|
* memory attribute setting for the physical pages cannot be configured
|
|
|
|
* to VM_MEMATTR_DEFAULT.
|
|
|
|
*
|
2016-12-28 18:32:13 +00:00
|
|
|
* The specified object may not contain fictitious pages.
|
|
|
|
*
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
* The caller must always specify an allocation class.
|
|
|
|
*
|
|
|
|
* allocation classes:
|
|
|
|
* VM_ALLOC_NORMAL normal process request
|
|
|
|
* VM_ALLOC_SYSTEM system *really* needs a page
|
|
|
|
* VM_ALLOC_INTERRUPT interrupt time request
|
|
|
|
*
|
|
|
|
* optional allocation flags:
|
2013-08-09 11:11:11 +00:00
|
|
|
* VM_ALLOC_NOBUSY do not exclusive busy the page
|
2015-05-30 23:37:47 +00:00
|
|
|
* VM_ALLOC_NODUMP do not include the page in a kernel core dump
|
2013-08-09 11:11:11 +00:00
|
|
|
* VM_ALLOC_SBUSY shared busy the allocated page
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
* VM_ALLOC_WIRED wire the allocated page
|
|
|
|
* VM_ALLOC_ZERO prefer a zeroed page
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_contig(vm_object_t object, vm_pindex_t pindex, int req,
|
|
|
|
u_long npages, vm_paddr_t low, vm_paddr_t high, u_long alignment,
|
|
|
|
vm_paddr_t boundary, vm_memattr_t memattr)
|
2017-11-28 23:18:35 +00:00
|
|
|
{
|
2018-01-12 22:48:23 +00:00
|
|
|
struct vm_domainset_iter di;
|
2017-11-28 23:18:35 +00:00
|
|
|
vm_page_t m;
|
2018-01-12 22:48:23 +00:00
|
|
|
int domain;
|
2017-11-28 23:18:35 +00:00
|
|
|
|
2018-03-29 02:54:50 +00:00
|
|
|
vm_domainset_iter_page_init(&di, object, pindex, &domain, &req);
|
2018-01-12 22:48:23 +00:00
|
|
|
do {
|
2017-11-28 23:18:35 +00:00
|
|
|
m = vm_page_alloc_contig_domain(object, pindex, domain, req,
|
|
|
|
npages, low, high, alignment, boundary, memattr);
|
|
|
|
if (m != NULL)
|
|
|
|
break;
|
2018-10-23 16:35:58 +00:00
|
|
|
} while (vm_domainset_iter_page(&di, object, &domain) == 0);
|
2017-11-28 23:18:35 +00:00
|
|
|
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
2021-12-24 04:45:47 +00:00
|
|
|
static vm_page_t
|
|
|
|
vm_page_find_contig_domain(int domain, int req, u_long npages, vm_paddr_t low,
|
|
|
|
vm_paddr_t high, u_long alignment, vm_paddr_t boundary)
|
|
|
|
{
|
|
|
|
struct vm_domain *vmd;
|
|
|
|
vm_page_t m_ret;
|
|
|
|
|
2021-12-24 22:10:30 +00:00
|
|
|
/*
|
|
|
|
* Can we allocate the pages without the number of free pages falling
|
|
|
|
* below the lower bound for the allocation class?
|
|
|
|
*/
|
2021-12-24 04:45:47 +00:00
|
|
|
vmd = VM_DOMAIN(domain);
|
2021-12-24 08:47:21 +00:00
|
|
|
if (!vm_domain_allocate(vmd, req, npages))
|
|
|
|
return (NULL);
|
2021-12-24 04:45:47 +00:00
|
|
|
/*
|
|
|
|
* Try to allocate the pages from the free page queues.
|
|
|
|
*/
|
|
|
|
vm_domain_free_lock(vmd);
|
|
|
|
m_ret = vm_phys_alloc_contig(domain, npages, low, high,
|
|
|
|
alignment, boundary);
|
|
|
|
vm_domain_free_unlock(vmd);
|
|
|
|
if (m_ret != NULL)
|
|
|
|
return (m_ret);
|
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
/*
|
2021-12-24 18:59:16 +00:00
|
|
|
* Try to break a reservation to allocate the pages.
|
2021-12-24 04:45:47 +00:00
|
|
|
*/
|
2021-12-24 18:59:16 +00:00
|
|
|
if ((req & VM_ALLOC_NORECLAIM) == 0) {
|
|
|
|
m_ret = vm_reserv_reclaim_contig(domain, npages, low,
|
|
|
|
high, alignment, boundary);
|
|
|
|
if (m_ret != NULL)
|
|
|
|
return (m_ret);
|
|
|
|
}
|
2021-12-24 04:45:47 +00:00
|
|
|
#endif
|
2021-12-24 18:59:16 +00:00
|
|
|
vm_domain_freecnt_inc(vmd, npages);
|
|
|
|
return (NULL);
|
2021-12-24 04:45:47 +00:00
|
|
|
}
|
|
|
|
|
2017-11-28 23:18:35 +00:00
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_contig_domain(vm_object_t object, vm_pindex_t pindex, int domain,
|
|
|
|
int req, u_long npages, vm_paddr_t low, vm_paddr_t high, u_long alignment,
|
|
|
|
vm_paddr_t boundary, vm_memattr_t memattr)
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
{
|
2016-12-28 18:32:13 +00:00
|
|
|
vm_page_t m, m_ret, mpred;
|
|
|
|
u_int busy_lock, flags, oflags;
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
|
2021-11-11 19:26:41 +00:00
|
|
|
#define VPAC_FLAGS (VPA_FLAGS | VM_ALLOC_NORECLAIM)
|
2021-10-20 00:26:09 +00:00
|
|
|
KASSERT((req & ~VPAC_FLAGS) == 0,
|
|
|
|
("invalid request %#x", req));
|
|
|
|
KASSERT(((req & (VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)) !=
|
2013-08-09 11:11:11 +00:00
|
|
|
(VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)),
|
2021-10-20 00:26:09 +00:00
|
|
|
("invalid request %#x", req));
|
2021-11-11 19:26:41 +00:00
|
|
|
KASSERT((req & (VM_ALLOC_WAITOK | VM_ALLOC_NORECLAIM)) !=
|
|
|
|
(VM_ALLOC_WAITOK | VM_ALLOC_NORECLAIM),
|
|
|
|
("invalid request %#x", req));
|
2021-10-20 00:26:09 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
|
|
|
KASSERT((object->flags & OBJ_FICTITIOUS) == 0,
|
|
|
|
("vm_page_alloc_contig: object %p has fictitious pages",
|
|
|
|
object));
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
KASSERT(npages > 0, ("vm_page_alloc_contig: npages is zero"));
|
|
|
|
|
2021-10-20 00:26:09 +00:00
|
|
|
mpred = vm_radix_lookup_le(&object->rtree, pindex);
|
|
|
|
KASSERT(mpred == NULL || mpred->pindex != pindex,
|
|
|
|
("vm_page_alloc_contig: pindex already allocated"));
|
2021-12-24 04:45:47 +00:00
|
|
|
for (;;) {
|
2018-02-06 22:10:07 +00:00
|
|
|
#if VM_NRESERVLEVEL > 0
|
2016-12-28 18:32:13 +00:00
|
|
|
/*
|
2021-12-24 04:45:47 +00:00
|
|
|
* Can we allocate the pages from a reservation?
|
2016-12-28 18:32:13 +00:00
|
|
|
*/
|
2021-12-24 04:45:47 +00:00
|
|
|
if (vm_object_reserv(object) &&
|
|
|
|
(m_ret = vm_reserv_alloc_contig(object, pindex, domain, req,
|
|
|
|
mpred, npages, low, high, alignment, boundary)) != NULL) {
|
|
|
|
break;
|
2018-03-22 19:21:11 +00:00
|
|
|
}
|
2018-02-06 22:10:07 +00:00
|
|
|
#endif
|
2021-12-24 04:45:47 +00:00
|
|
|
if ((m_ret = vm_page_find_contig_domain(domain, req, npages,
|
|
|
|
low, high, alignment, boundary)) != NULL)
|
|
|
|
break;
|
|
|
|
if (!vm_domain_alloc_fail(VM_DOMAIN(domain), object, req))
|
|
|
|
return (NULL);
|
|
|
|
}
|
2018-04-24 21:15:54 +00:00
|
|
|
for (m = m_ret; m < &m_ret[npages]; m++) {
|
|
|
|
vm_page_dequeue(m);
|
2016-11-15 18:22:50 +00:00
|
|
|
vm_page_alloc_check(m);
|
2018-04-24 21:15:54 +00:00
|
|
|
}
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Initialize the pages. Only the PG_ZERO flag is inherited.
|
|
|
|
*/
|
2021-10-20 00:26:09 +00:00
|
|
|
flags = PG_ZERO;
|
2012-01-27 20:18:31 +00:00
|
|
|
if ((req & VM_ALLOC_NODUMP) != 0)
|
|
|
|
flags |= PG_NODUMP;
|
2021-10-20 00:26:09 +00:00
|
|
|
oflags = (object->flags & OBJ_UNMANAGED) != 0 ? VPO_UNMANAGED : 0;
|
|
|
|
if ((req & (VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)) == 0)
|
2019-11-24 19:12:23 +00:00
|
|
|
busy_lock = VPB_CURTHREAD_EXCLUSIVE;
|
2020-02-04 20:33:01 +00:00
|
|
|
else if ((req & VM_ALLOC_SBUSY) != 0)
|
2016-12-28 18:32:13 +00:00
|
|
|
busy_lock = VPB_SHARERS_WORD(1);
|
2020-02-04 20:33:01 +00:00
|
|
|
else
|
|
|
|
busy_lock = VPB_UNBUSIED;
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
if ((req & VM_ALLOC_WIRED) != 0)
|
2018-02-12 22:53:00 +00:00
|
|
|
vm_wire_add(npages);
|
2021-10-20 00:26:09 +00:00
|
|
|
if (object->memattr != VM_MEMATTR_DEFAULT &&
|
|
|
|
memattr == VM_MEMATTR_DEFAULT)
|
|
|
|
memattr = object->memattr;
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
for (m = m_ret; m < &m_ret[npages]; m++) {
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.flags = 0;
|
2012-07-17 02:36:59 +00:00
|
|
|
m->flags = (m->flags | PG_NODUMP) & flags;
|
2016-12-28 18:32:13 +00:00
|
|
|
m->busy_lock = busy_lock;
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
if ((req & VM_ALLOC_WIRED) != 0)
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
m->ref_count = 1;
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.act_count = 0;
|
2016-12-28 18:32:13 +00:00
|
|
|
m->oflags = oflags;
|
2021-10-20 00:26:09 +00:00
|
|
|
if (vm_page_insert_after(m, object, pindex, mpred)) {
|
|
|
|
if ((req & VM_ALLOC_WIRED) != 0)
|
|
|
|
vm_wire_sub(npages);
|
|
|
|
KASSERT(m->object == NULL,
|
|
|
|
("page %p has object", m));
|
2016-12-28 18:32:13 +00:00
|
|
|
mpred = m;
|
2021-10-20 00:26:09 +00:00
|
|
|
for (m = m_ret; m < &m_ret[npages]; m++) {
|
|
|
|
if (m <= mpred &&
|
|
|
|
(req & VM_ALLOC_WIRED) != 0)
|
|
|
|
m->ref_count = 0;
|
|
|
|
m->oflags = VPO_UNMANAGED;
|
|
|
|
m->busy_lock = VPB_UNBUSIED;
|
|
|
|
/* Don't change PG_ZERO. */
|
|
|
|
vm_page_free_toq(m);
|
|
|
|
}
|
|
|
|
if (req & VM_ALLOC_WAITFAIL) {
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
vm_radix_wait();
|
|
|
|
VM_OBJECT_WLOCK(object);
|
|
|
|
}
|
|
|
|
return (NULL);
|
|
|
|
}
|
|
|
|
mpred = m;
|
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.
From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.
Reviewed by: kib
Discussed with: jhb
2011-11-16 16:46:09 +00:00
|
|
|
if (memattr != VM_MEMATTR_DEFAULT)
|
|
|
|
pmap_page_set_memattr(m, memattr);
|
|
|
|
pindex++;
|
|
|
|
}
|
|
|
|
return (m_ret);
|
|
|
|
}
|
|
|
|
|
Redo the page table page allocation on MIPS, as suggested by
alc@.
The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.
This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).
The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().
Reviewed by: alc
2010-07-21 09:27:00 +00:00
|
|
|
/*
|
2021-10-20 00:22:12 +00:00
|
|
|
* Allocate a physical page that is not intended to be inserted into a VM
|
|
|
|
* object. If the "freelist" parameter is not equal to VM_NFREELIST, then only
|
|
|
|
* pages from the specified vm_phys freelist will be returned.
|
Redo the page table page allocation on MIPS, as suggested by
alc@.
The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.
This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).
The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().
Reviewed by: alc
2010-07-21 09:27:00 +00:00
|
|
|
*/
|
2021-10-20 00:22:12 +00:00
|
|
|
static __always_inline vm_page_t
|
|
|
|
_vm_page_alloc_noobj_domain(int domain, const int freelist, int req)
|
Redo the page table page allocation on MIPS, as suggested by
alc@.
The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.
This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).
The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().
Reviewed by: alc
2010-07-21 09:27:00 +00:00
|
|
|
{
|
2018-02-06 22:10:07 +00:00
|
|
|
struct vm_domain *vmd;
|
Redo the page table page allocation on MIPS, as suggested by
alc@.
The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.
This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).
The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().
Reviewed by: alc
2010-07-21 09:27:00 +00:00
|
|
|
vm_page_t m;
|
2021-10-20 00:22:12 +00:00
|
|
|
int flags;
|
Redo the page table page allocation on MIPS, as suggested by
alc@.
The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.
This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).
The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().
Reviewed by: alc
2010-07-21 09:27:00 +00:00
|
|
|
|
2021-10-20 00:25:14 +00:00
|
|
|
#define VPAN_FLAGS (VM_ALLOC_CLASS_MASK | VM_ALLOC_WAITFAIL | \
|
|
|
|
VM_ALLOC_NOWAIT | VM_ALLOC_WAITOK | \
|
|
|
|
VM_ALLOC_NOBUSY | VM_ALLOC_WIRED | \
|
|
|
|
VM_ALLOC_NODUMP | VM_ALLOC_ZERO | VM_ALLOC_COUNT_MASK)
|
|
|
|
KASSERT((req & ~VPAN_FLAGS) == 0,
|
|
|
|
("invalid request %#x", req));
|
2021-10-20 00:22:12 +00:00
|
|
|
|
|
|
|
flags = (req & VM_ALLOC_NODUMP) != 0 ? PG_NODUMP : 0;
|
2018-02-06 22:10:07 +00:00
|
|
|
vmd = VM_DOMAIN(domain);
|
2017-11-08 02:39:37 +00:00
|
|
|
again:
|
2021-10-20 00:22:12 +00:00
|
|
|
if (freelist == VM_NFREELIST &&
|
|
|
|
vmd->vmd_pgcache[VM_FREEPOOL_DIRECT].zone != NULL) {
|
|
|
|
m = uma_zalloc(vmd->vmd_pgcache[VM_FREEPOOL_DIRECT].zone,
|
|
|
|
M_NOWAIT | M_NOVM);
|
|
|
|
if (m != NULL) {
|
|
|
|
flags |= PG_PCPU_CACHE;
|
|
|
|
goto found;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-03-22 19:21:11 +00:00
|
|
|
if (vm_domain_allocate(vmd, req, 1)) {
|
|
|
|
vm_domain_free_lock(vmd);
|
2021-10-20 00:22:12 +00:00
|
|
|
if (freelist == VM_NFREELIST)
|
|
|
|
m = vm_phys_alloc_pages(domain, VM_FREEPOOL_DIRECT, 0);
|
|
|
|
else
|
|
|
|
m = vm_phys_alloc_freelist_pages(domain, freelist,
|
|
|
|
VM_FREEPOOL_DIRECT, 0);
|
2018-03-22 19:21:11 +00:00
|
|
|
vm_domain_free_unlock(vmd);
|
2021-10-21 15:46:25 +00:00
|
|
|
if (m == NULL) {
|
2018-03-22 19:21:11 +00:00
|
|
|
vm_domain_freecnt_inc(vmd, 1);
|
2021-10-21 15:46:25 +00:00
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
if (freelist == VM_NFREELIST &&
|
|
|
|
vm_reserv_reclaim_inactive(domain))
|
|
|
|
goto again;
|
|
|
|
#endif
|
|
|
|
}
|
2018-03-22 19:21:11 +00:00
|
|
|
}
|
2017-11-28 23:18:35 +00:00
|
|
|
if (m == NULL) {
|
2018-02-06 22:10:07 +00:00
|
|
|
if (vm_domain_alloc_fail(vmd, NULL, req))
|
2017-11-08 02:39:37 +00:00
|
|
|
goto again;
|
2011-11-06 02:03:27 +00:00
|
|
|
return (NULL);
|
Redo the page table page allocation on MIPS, as suggested by
alc@.
The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.
This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).
The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().
Reviewed by: alc
2010-07-21 09:27:00 +00:00
|
|
|
}
|
2021-10-20 00:22:12 +00:00
|
|
|
|
|
|
|
found:
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_page_dequeue(m);
|
2016-11-15 18:22:50 +00:00
|
|
|
vm_page_alloc_check(m);
|
2011-11-02 05:42:51 +00:00
|
|
|
|
2021-10-20 00:25:14 +00:00
|
|
|
/*
|
|
|
|
* Consumers should not rely on a useful default pindex value.
|
|
|
|
*/
|
2021-10-20 00:22:12 +00:00
|
|
|
m->pindex = 0xdeadc0dedeadc0de;
|
|
|
|
m->flags = (m->flags & PG_ZERO) | flags;
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.flags = 0;
|
2021-10-20 00:22:12 +00:00
|
|
|
m->oflags = VPO_UNMANAGED;
|
|
|
|
m->busy_lock = VPB_UNBUSIED;
|
2011-11-02 05:42:51 +00:00
|
|
|
if ((req & VM_ALLOC_WIRED) != 0) {
|
2018-02-12 22:53:00 +00:00
|
|
|
vm_wire_add(1);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
m->ref_count = 1;
|
2011-11-02 05:42:51 +00:00
|
|
|
}
|
2021-10-20 00:22:12 +00:00
|
|
|
|
|
|
|
if ((req & VM_ALLOC_ZERO) != 0 && (m->flags & PG_ZERO) == 0)
|
|
|
|
pmap_zero_page(m);
|
|
|
|
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_freelist(int freelist, int req)
|
|
|
|
{
|
|
|
|
struct vm_domainset_iter di;
|
|
|
|
vm_page_t m;
|
|
|
|
int domain;
|
|
|
|
|
|
|
|
vm_domainset_iter_page_init(&di, NULL, 0, &domain, &req);
|
|
|
|
do {
|
|
|
|
m = vm_page_alloc_freelist_domain(domain, freelist, req);
|
|
|
|
if (m != NULL)
|
|
|
|
break;
|
|
|
|
} while (vm_domainset_iter_page(&di, NULL, &domain) == 0);
|
|
|
|
|
Redo the page table page allocation on MIPS, as suggested by
alc@.
The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.
This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).
The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().
Reviewed by: alc
2010-07-21 09:27:00 +00:00
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
2021-10-20 00:22:12 +00:00
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_freelist_domain(int domain, int freelist, int req)
|
|
|
|
{
|
|
|
|
KASSERT(freelist >= 0 && freelist < VM_NFREELIST,
|
|
|
|
("%s: invalid freelist %d", __func__, freelist));
|
|
|
|
|
|
|
|
return (_vm_page_alloc_noobj_domain(domain, freelist, req));
|
|
|
|
}
|
|
|
|
|
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_noobj(int req)
|
|
|
|
{
|
|
|
|
struct vm_domainset_iter di;
|
|
|
|
vm_page_t m;
|
|
|
|
int domain;
|
|
|
|
|
|
|
|
vm_domainset_iter_page_init(&di, NULL, 0, &domain, &req);
|
|
|
|
do {
|
|
|
|
m = vm_page_alloc_noobj_domain(domain, req);
|
|
|
|
if (m != NULL)
|
|
|
|
break;
|
|
|
|
} while (vm_domainset_iter_page(&di, NULL, &domain) == 0);
|
|
|
|
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_noobj_domain(int domain, int req)
|
|
|
|
{
|
|
|
|
return (_vm_page_alloc_noobj_domain(domain, VM_NFREELIST, req));
|
|
|
|
}
|
|
|
|
|
2021-10-20 00:24:21 +00:00
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_noobj_contig(int req, u_long npages, vm_paddr_t low,
|
|
|
|
vm_paddr_t high, u_long alignment, vm_paddr_t boundary,
|
|
|
|
vm_memattr_t memattr)
|
|
|
|
{
|
|
|
|
struct vm_domainset_iter di;
|
|
|
|
vm_page_t m;
|
|
|
|
int domain;
|
|
|
|
|
|
|
|
vm_domainset_iter_page_init(&di, NULL, 0, &domain, &req);
|
|
|
|
do {
|
|
|
|
m = vm_page_alloc_noobj_contig_domain(domain, req, npages, low,
|
|
|
|
high, alignment, boundary, memattr);
|
|
|
|
if (m != NULL)
|
|
|
|
break;
|
|
|
|
} while (vm_domainset_iter_page(&di, NULL, &domain) == 0);
|
|
|
|
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
|
|
|
vm_page_t
|
|
|
|
vm_page_alloc_noobj_contig_domain(int domain, int req, u_long npages,
|
|
|
|
vm_paddr_t low, vm_paddr_t high, u_long alignment, vm_paddr_t boundary,
|
|
|
|
vm_memattr_t memattr)
|
|
|
|
{
|
2021-10-20 00:26:09 +00:00
|
|
|
vm_page_t m, m_ret;
|
|
|
|
u_int flags;
|
|
|
|
|
2021-11-11 19:26:41 +00:00
|
|
|
#define VPANC_FLAGS (VPAN_FLAGS | VM_ALLOC_NORECLAIM)
|
2021-10-20 00:26:09 +00:00
|
|
|
KASSERT((req & ~VPANC_FLAGS) == 0,
|
|
|
|
("invalid request %#x", req));
|
2021-11-11 19:26:41 +00:00
|
|
|
KASSERT((req & (VM_ALLOC_WAITOK | VM_ALLOC_NORECLAIM)) !=
|
|
|
|
(VM_ALLOC_WAITOK | VM_ALLOC_NORECLAIM),
|
|
|
|
("invalid request %#x", req));
|
2021-10-20 00:26:09 +00:00
|
|
|
KASSERT(((req & (VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)) !=
|
|
|
|
(VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY)),
|
|
|
|
("invalid request %#x", req));
|
|
|
|
KASSERT(npages > 0, ("vm_page_alloc_contig: npages is zero"));
|
|
|
|
|
2021-12-24 04:45:47 +00:00
|
|
|
while ((m_ret = vm_page_find_contig_domain(domain, req, npages,
|
|
|
|
low, high, alignment, boundary)) == NULL) {
|
|
|
|
if (!vm_domain_alloc_fail(VM_DOMAIN(domain), NULL, req))
|
|
|
|
return (NULL);
|
2021-10-20 00:26:09 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Initialize the pages. Only the PG_ZERO flag is inherited.
|
|
|
|
*/
|
|
|
|
flags = PG_ZERO;
|
|
|
|
if ((req & VM_ALLOC_NODUMP) != 0)
|
|
|
|
flags |= PG_NODUMP;
|
|
|
|
if ((req & VM_ALLOC_WIRED) != 0)
|
|
|
|
vm_wire_add(npages);
|
|
|
|
for (m = m_ret; m < &m_ret[npages]; m++) {
|
|
|
|
vm_page_dequeue(m);
|
|
|
|
vm_page_alloc_check(m);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Consumers should not rely on a useful default pindex value.
|
|
|
|
*/
|
|
|
|
m->pindex = 0xdeadc0dedeadc0de;
|
|
|
|
m->a.flags = 0;
|
|
|
|
m->flags = (m->flags | PG_NODUMP) & flags;
|
|
|
|
m->busy_lock = VPB_UNBUSIED;
|
|
|
|
if ((req & VM_ALLOC_WIRED) != 0)
|
|
|
|
m->ref_count = 1;
|
|
|
|
m->a.act_count = 0;
|
|
|
|
m->oflags = VPO_UNMANAGED;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Zero the page before updating any mappings since the page is
|
|
|
|
* not yet shared with any devices which might require the
|
|
|
|
* non-default memory attribute. pmap_page_set_memattr()
|
|
|
|
* flushes data caches before returning.
|
|
|
|
*/
|
|
|
|
if ((req & VM_ALLOC_ZERO) != 0 && (m->flags & PG_ZERO) == 0)
|
|
|
|
pmap_zero_page(m);
|
|
|
|
if (memattr != VM_MEMATTR_DEFAULT)
|
|
|
|
pmap_page_set_memattr(m, memattr);
|
|
|
|
}
|
|
|
|
return (m_ret);
|
2021-10-20 00:24:21 +00:00
|
|
|
}
|
|
|
|
|
2021-10-20 01:22:50 +00:00
|
|
|
/*
|
|
|
|
* Check a page that has been freshly dequeued from a freelist.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
vm_page_alloc_check(vm_page_t m)
|
|
|
|
{
|
2021-10-20 00:22:12 +00:00
|
|
|
|
2021-10-20 01:22:50 +00:00
|
|
|
KASSERT(m->object == NULL, ("page %p has object", m));
|
|
|
|
KASSERT(m->a.queue == PQ_NONE &&
|
|
|
|
(m->a.flags & PGA_QUEUE_STATE_MASK) == 0,
|
|
|
|
("page %p has unexpected queue %d, flags %#x",
|
|
|
|
m, m->a.queue, (m->a.flags & PGA_QUEUE_STATE_MASK)));
|
2021-10-20 00:22:12 +00:00
|
|
|
KASSERT(m->ref_count == 0, ("page %p has references", m));
|
|
|
|
KASSERT(vm_page_busy_freed(m), ("page %p is not freed", m));
|
2021-10-20 01:22:50 +00:00
|
|
|
KASSERT(m->dirty == 0, ("page %p is dirty", m));
|
|
|
|
KASSERT(pmap_page_get_memattr(m) == VM_MEMATTR_DEFAULT,
|
|
|
|
("page %p has unexpected memattr %d",
|
|
|
|
m, pmap_page_get_memattr(m)));
|
2022-10-18 09:16:36 +00:00
|
|
|
KASSERT(vm_page_none_valid(m), ("free page %p is valid", m));
|
2021-10-20 01:22:50 +00:00
|
|
|
pmap_vm_page_alloc_check(m);
|
|
|
|
}
|
|
|
|
|
2018-04-01 04:50:05 +00:00
|
|
|
static int
|
2019-07-20 18:25:41 +00:00
|
|
|
vm_page_zone_import(void *arg, void **store, int cnt, int domain, int flags)
|
2018-04-01 04:50:05 +00:00
|
|
|
{
|
|
|
|
struct vm_domain *vmd;
|
2019-07-08 18:56:30 +00:00
|
|
|
struct vm_pgcache *pgcache;
|
2018-06-26 18:29:56 +00:00
|
|
|
int i;
|
2018-04-01 04:50:05 +00:00
|
|
|
|
2019-07-08 18:56:30 +00:00
|
|
|
pgcache = arg;
|
|
|
|
vmd = VM_DOMAIN(pgcache->domain);
|
2019-11-22 16:31:10 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The page daemon should avoid creating extra memory pressure since its
|
|
|
|
* main purpose is to replenish the store of free pages.
|
|
|
|
*/
|
|
|
|
if (vmd->vmd_severeset || curproc == pageproc ||
|
2020-08-12 00:21:30 +00:00
|
|
|
!_vm_domain_allocate(vmd, VM_ALLOC_NORMAL, cnt))
|
2018-04-01 04:50:05 +00:00
|
|
|
return (0);
|
|
|
|
domain = vmd->vmd_domain;
|
|
|
|
vm_domain_free_lock(vmd);
|
2019-07-08 18:56:30 +00:00
|
|
|
i = vm_phys_alloc_npages(domain, pgcache->pool, cnt,
|
2018-06-26 18:29:56 +00:00
|
|
|
(vm_page_t *)store);
|
2018-04-01 04:50:05 +00:00
|
|
|
vm_domain_free_unlock(vmd);
|
|
|
|
if (cnt != i)
|
|
|
|
vm_domain_freecnt_inc(vmd, cnt - i);
|
|
|
|
|
|
|
|
return (i);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
2019-07-20 18:25:41 +00:00
|
|
|
vm_page_zone_release(void *arg, void **store, int cnt)
|
2018-04-01 04:50:05 +00:00
|
|
|
{
|
|
|
|
struct vm_domain *vmd;
|
2019-07-08 18:56:30 +00:00
|
|
|
struct vm_pgcache *pgcache;
|
2018-04-01 04:50:05 +00:00
|
|
|
vm_page_t m;
|
|
|
|
int i;
|
|
|
|
|
2019-07-08 18:56:30 +00:00
|
|
|
pgcache = arg;
|
|
|
|
vmd = VM_DOMAIN(pgcache->domain);
|
2018-04-01 04:50:05 +00:00
|
|
|
vm_domain_free_lock(vmd);
|
|
|
|
for (i = 0; i < cnt; i++) {
|
|
|
|
m = (vm_page_t)store[i];
|
|
|
|
vm_phys_free_pages(m, 0);
|
|
|
|
}
|
|
|
|
vm_domain_free_unlock(vmd);
|
|
|
|
vm_domain_freecnt_inc(vmd, cnt);
|
|
|
|
}
|
|
|
|
|
2015-12-19 18:42:50 +00:00
|
|
|
#define VPSC_ANY 0 /* No restrictions. */
|
|
|
|
#define VPSC_NORESERV 1 /* Skip reservations; implies VPSC_NOSUPER. */
|
|
|
|
#define VPSC_NOSUPER 2 /* Skip superpages. */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_scan_contig:
|
|
|
|
*
|
|
|
|
* Scan vm_page_array[] between the specified entries "m_start" and
|
|
|
|
* "m_end" for a run of contiguous physical pages that satisfy the
|
|
|
|
* specified conditions, and return the lowest page in the run. The
|
|
|
|
* specified "alignment" determines the alignment of the lowest physical
|
|
|
|
* page in the run. If the specified "boundary" is non-zero, then the
|
|
|
|
* run of physical pages cannot span a physical address that is a
|
|
|
|
* multiple of "boundary".
|
|
|
|
*
|
|
|
|
* "m_end" is never dereferenced, so it need not point to a vm_page
|
|
|
|
* structure within vm_page_array[].
|
|
|
|
*
|
|
|
|
* "npages" must be greater than zero. "m_start" and "m_end" must not
|
|
|
|
* span a hole (or discontiguity) in the physical address space. Both
|
|
|
|
* "alignment" and "boundary" must be a power of two.
|
|
|
|
*/
|
2023-06-16 06:43:45 +00:00
|
|
|
static vm_page_t
|
2015-12-19 18:42:50 +00:00
|
|
|
vm_page_scan_contig(u_long npages, vm_page_t m_start, vm_page_t m_end,
|
|
|
|
u_long alignment, vm_paddr_t boundary, int options)
|
|
|
|
{
|
|
|
|
vm_object_t object;
|
|
|
|
vm_paddr_t pa;
|
|
|
|
vm_page_t m, m_run;
|
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
int level;
|
|
|
|
#endif
|
|
|
|
int m_inc, order, run_ext, run_len;
|
|
|
|
|
|
|
|
KASSERT(npages > 0, ("npages is 0"));
|
|
|
|
KASSERT(powerof2(alignment), ("alignment is not a power of 2"));
|
|
|
|
KASSERT(powerof2(boundary), ("boundary is not a power of 2"));
|
|
|
|
m_run = NULL;
|
|
|
|
run_len = 0;
|
|
|
|
for (m = m_start; m < m_end && run_len < npages; m += m_inc) {
|
2017-11-21 13:17:40 +00:00
|
|
|
KASSERT((m->flags & PG_MARKER) == 0,
|
|
|
|
("page %p is PG_MARKER", m));
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT((m->flags & PG_FICTITIOUS) == 0 || m->ref_count >= 1,
|
|
|
|
("fictitious page %p has invalid ref count", m));
|
2015-12-19 18:42:50 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the current page would be the start of a run, check its
|
|
|
|
* physical address against the end, alignment, and boundary
|
|
|
|
* conditions. If it doesn't satisfy these conditions, either
|
|
|
|
* terminate the scan or advance to the next page that
|
|
|
|
* satisfies the failed condition.
|
|
|
|
*/
|
|
|
|
if (run_len == 0) {
|
|
|
|
KASSERT(m_run == NULL, ("m_run != NULL"));
|
|
|
|
if (m + npages > m_end)
|
|
|
|
break;
|
|
|
|
pa = VM_PAGE_TO_PHYS(m);
|
2021-12-31 04:09:08 +00:00
|
|
|
if (!vm_addr_align_ok(pa, alignment)) {
|
2015-12-19 18:42:50 +00:00
|
|
|
m_inc = atop(roundup2(pa, alignment) - pa);
|
|
|
|
continue;
|
|
|
|
}
|
2021-12-31 04:09:08 +00:00
|
|
|
if (!vm_addr_bound_ok(pa, ptoa(npages), boundary)) {
|
2015-12-19 18:42:50 +00:00
|
|
|
m_inc = atop(roundup2(pa, boundary) - pa);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
KASSERT(m_run != NULL, ("m_run == NULL"));
|
|
|
|
|
|
|
|
retry:
|
2020-02-01 18:23:51 +00:00
|
|
|
m_inc = 1;
|
2019-07-08 19:46:20 +00:00
|
|
|
if (vm_page_wired(m))
|
2015-12-19 18:42:50 +00:00
|
|
|
run_ext = 0;
|
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
else if ((level = vm_reserv_level(m)) >= 0 &&
|
|
|
|
(options & VPSC_NORESERV) != 0) {
|
|
|
|
run_ext = 0;
|
|
|
|
/* Advance to the end of the reservation. */
|
|
|
|
pa = VM_PAGE_TO_PHYS(m);
|
|
|
|
m_inc = atop(roundup2(pa + 1, vm_reserv_size(level)) -
|
|
|
|
pa);
|
|
|
|
}
|
|
|
|
#endif
|
2020-02-14 23:16:29 +00:00
|
|
|
else if ((object = atomic_load_ptr(&m->object)) != NULL) {
|
2015-12-19 18:42:50 +00:00
|
|
|
/*
|
|
|
|
* The page is considered eligible for relocation if
|
|
|
|
* and only if it could be laundered or reclaimed by
|
|
|
|
* the page daemon.
|
|
|
|
*/
|
2020-02-01 18:23:51 +00:00
|
|
|
VM_OBJECT_RLOCK(object);
|
|
|
|
if (object != m->object) {
|
|
|
|
VM_OBJECT_RUNLOCK(object);
|
|
|
|
goto retry;
|
2015-12-19 18:42:50 +00:00
|
|
|
}
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
/* Don't care: PG_NODUMP, PG_ZERO. */
|
2022-07-16 15:29:19 +00:00
|
|
|
if ((object->flags & OBJ_SWAP) == 0 &&
|
2016-11-15 18:22:50 +00:00
|
|
|
object->type != OBJT_VNODE) {
|
2015-12-19 18:42:50 +00:00
|
|
|
run_ext = 0;
|
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
} else if ((options & VPSC_NOSUPER) != 0 &&
|
|
|
|
(level = vm_reserv_level_iffullpop(m)) >= 0) {
|
|
|
|
run_ext = 0;
|
|
|
|
/* Advance to the end of the superpage. */
|
|
|
|
pa = VM_PAGE_TO_PHYS(m);
|
|
|
|
m_inc = atop(roundup2(pa + 1,
|
|
|
|
vm_reserv_size(level)) - pa);
|
|
|
|
#endif
|
|
|
|
} else if (object->memattr == VM_MEMATTR_DEFAULT &&
|
2020-02-01 18:23:51 +00:00
|
|
|
vm_page_queue(m) != PQ_NONE && !vm_page_busied(m)) {
|
2015-12-19 18:42:50 +00:00
|
|
|
/*
|
|
|
|
* The page is allocated but eligible for
|
|
|
|
* relocation. Extend the current run by one
|
|
|
|
* page.
|
|
|
|
*/
|
|
|
|
KASSERT(pmap_page_get_memattr(m) ==
|
|
|
|
VM_MEMATTR_DEFAULT,
|
|
|
|
("page %p has an unexpected memattr", m));
|
|
|
|
KASSERT((m->oflags & (VPO_SWAPINPROG |
|
|
|
|
VPO_SWAPSLEEP | VPO_UNMANAGED)) == 0,
|
|
|
|
("page %p has unexpected oflags", m));
|
2019-10-15 03:48:22 +00:00
|
|
|
/* Don't care: PGA_NOSYNC. */
|
2015-12-19 18:42:50 +00:00
|
|
|
run_ext = 1;
|
|
|
|
} else
|
|
|
|
run_ext = 0;
|
|
|
|
VM_OBJECT_RUNLOCK(object);
|
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
} else if (level >= 0) {
|
|
|
|
/*
|
|
|
|
* The page is reserved but not yet allocated. In
|
2016-12-12 17:47:09 +00:00
|
|
|
* other words, it is still free. Extend the current
|
|
|
|
* run by one page.
|
2015-12-19 18:42:50 +00:00
|
|
|
*/
|
|
|
|
run_ext = 1;
|
|
|
|
#endif
|
|
|
|
} else if ((order = m->order) < VM_NFREEORDER) {
|
|
|
|
/*
|
|
|
|
* The page is enqueued in the physical memory
|
2016-12-12 17:47:09 +00:00
|
|
|
* allocator's free page queues. Moreover, it is the
|
|
|
|
* first page in a power-of-two-sized run of
|
|
|
|
* contiguous free pages. Add these pages to the end
|
|
|
|
* of the current run, and jump ahead.
|
2015-12-19 18:42:50 +00:00
|
|
|
*/
|
|
|
|
run_ext = 1 << order;
|
|
|
|
m_inc = 1 << order;
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Skip the page for one of the following reasons: (1)
|
|
|
|
* It is enqueued in the physical memory allocator's
|
2016-12-12 17:47:09 +00:00
|
|
|
* free page queues. However, it is not the first
|
|
|
|
* page in a run of contiguous free pages. (This case
|
|
|
|
* rarely occurs because the scan is performed in
|
|
|
|
* ascending order.) (2) It is not reserved, and it is
|
|
|
|
* transitioning from free to allocated. (Conversely,
|
|
|
|
* the transition from allocated to free for managed
|
2020-08-04 14:59:43 +00:00
|
|
|
* pages is blocked by the page busy lock.) (3) It is
|
2016-12-12 17:47:09 +00:00
|
|
|
* allocated but not contained by an object and not
|
|
|
|
* wired, e.g., allocated by Xen's balloon driver.
|
2015-12-19 18:42:50 +00:00
|
|
|
*/
|
|
|
|
run_ext = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Extend or reset the current run of pages.
|
|
|
|
*/
|
|
|
|
if (run_ext > 0) {
|
|
|
|
if (run_len == 0)
|
|
|
|
m_run = m;
|
|
|
|
run_len += run_ext;
|
|
|
|
} else {
|
|
|
|
if (run_len > 0) {
|
|
|
|
m_run = NULL;
|
|
|
|
run_len = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (run_len >= npages)
|
|
|
|
return (m_run);
|
|
|
|
return (NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_reclaim_run:
|
|
|
|
*
|
|
|
|
* Try to relocate each of the allocated virtual pages within the
|
|
|
|
* specified run of physical pages to a new physical address. Free the
|
|
|
|
* physical pages underlying the relocated virtual pages. A virtual page
|
|
|
|
* is relocatable if and only if it could be laundered or reclaimed by
|
|
|
|
* the page daemon. Whenever possible, a virtual page is relocated to a
|
|
|
|
* physical address above "high".
|
|
|
|
*
|
|
|
|
* Returns 0 if every physical page within the run was already free or
|
|
|
|
* just freed by a successful relocation. Otherwise, returns a non-zero
|
|
|
|
* value indicating why the last attempt to relocate a virtual page was
|
|
|
|
* unsuccessful.
|
|
|
|
*
|
|
|
|
* "req_class" must be an allocation class.
|
|
|
|
*/
|
|
|
|
static int
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_page_reclaim_run(int req_class, int domain, u_long npages, vm_page_t m_run,
|
2015-12-19 18:42:50 +00:00
|
|
|
vm_paddr_t high)
|
|
|
|
{
|
2018-02-06 22:10:07 +00:00
|
|
|
struct vm_domain *vmd;
|
2015-12-19 18:42:50 +00:00
|
|
|
struct spglist free;
|
|
|
|
vm_object_t object;
|
|
|
|
vm_paddr_t pa;
|
|
|
|
vm_page_t m, m_end, m_new;
|
|
|
|
int error, order, req;
|
|
|
|
|
|
|
|
KASSERT((req_class & VM_ALLOC_CLASS_MASK) == req_class,
|
|
|
|
("req_class is not an allocation class"));
|
|
|
|
SLIST_INIT(&free);
|
|
|
|
error = 0;
|
|
|
|
m = m_run;
|
|
|
|
m_end = m_run + npages;
|
|
|
|
for (; error == 0 && m < m_end; m++) {
|
|
|
|
KASSERT((m->flags & (PG_FICTITIOUS | PG_MARKER)) == 0,
|
|
|
|
("page %p is PG_FICTITIOUS or PG_MARKER", m));
|
|
|
|
|
|
|
|
/*
|
2020-02-01 18:23:51 +00:00
|
|
|
* Racily check for wirings. Races are handled once the object
|
|
|
|
* lock is held and the page is unmapped.
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
*/
|
2019-07-08 19:46:20 +00:00
|
|
|
if (vm_page_wired(m))
|
2015-12-19 18:42:50 +00:00
|
|
|
error = EBUSY;
|
2020-02-14 23:16:29 +00:00
|
|
|
else if ((object = atomic_load_ptr(&m->object)) != NULL) {
|
2015-12-19 18:42:50 +00:00
|
|
|
/*
|
|
|
|
* The page is relocated if and only if it could be
|
|
|
|
* laundered or reclaimed by the page daemon.
|
|
|
|
*/
|
2020-02-01 18:23:51 +00:00
|
|
|
VM_OBJECT_WLOCK(object);
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
/* Don't care: PG_NODUMP, PG_ZERO. */
|
2020-02-01 18:23:51 +00:00
|
|
|
if (m->object != object ||
|
2022-07-16 15:29:19 +00:00
|
|
|
((object->flags & OBJ_SWAP) == 0 &&
|
2020-02-01 18:23:51 +00:00
|
|
|
object->type != OBJT_VNODE))
|
2015-12-19 18:42:50 +00:00
|
|
|
error = EINVAL;
|
2016-11-15 18:22:50 +00:00
|
|
|
else if (object->memattr != VM_MEMATTR_DEFAULT)
|
2015-12-19 18:42:50 +00:00
|
|
|
error = EINVAL;
|
2019-09-16 15:04:45 +00:00
|
|
|
else if (vm_page_queue(m) != PQ_NONE &&
|
2019-10-15 03:35:11 +00:00
|
|
|
vm_page_tryxbusy(m) != 0) {
|
|
|
|
if (vm_page_wired(m)) {
|
|
|
|
vm_page_xunbusy(m);
|
|
|
|
error = EBUSY;
|
|
|
|
goto unlock;
|
|
|
|
}
|
2015-12-19 18:42:50 +00:00
|
|
|
KASSERT(pmap_page_get_memattr(m) ==
|
|
|
|
VM_MEMATTR_DEFAULT,
|
|
|
|
("page %p has an unexpected memattr", m));
|
2019-12-21 19:04:05 +00:00
|
|
|
KASSERT(m->oflags == 0,
|
2015-12-19 18:42:50 +00:00
|
|
|
("page %p has unexpected oflags", m));
|
2019-10-15 03:48:22 +00:00
|
|
|
/* Don't care: PGA_NOSYNC. */
|
2019-10-15 03:45:41 +00:00
|
|
|
if (!vm_page_none_valid(m)) {
|
2015-12-19 18:42:50 +00:00
|
|
|
/*
|
|
|
|
* First, try to allocate a new page
|
|
|
|
* that is above "high". Failing
|
|
|
|
* that, try to allocate a new page
|
|
|
|
* that is below "m_run". Allocate
|
|
|
|
* the new page between the end of
|
|
|
|
* "m_run" and "high" only as a last
|
|
|
|
* resort.
|
|
|
|
*/
|
2021-10-20 00:25:04 +00:00
|
|
|
req = req_class;
|
2015-12-19 18:42:50 +00:00
|
|
|
if ((m->flags & PG_NODUMP) != 0)
|
|
|
|
req |= VM_ALLOC_NODUMP;
|
|
|
|
if (trunc_page(high) !=
|
|
|
|
~(vm_paddr_t)PAGE_MASK) {
|
2021-10-20 00:25:04 +00:00
|
|
|
m_new =
|
|
|
|
vm_page_alloc_noobj_contig(
|
|
|
|
req, 1, round_page(high),
|
|
|
|
~(vm_paddr_t)0, PAGE_SIZE,
|
|
|
|
0, VM_MEMATTR_DEFAULT);
|
2015-12-19 18:42:50 +00:00
|
|
|
} else
|
|
|
|
m_new = NULL;
|
|
|
|
if (m_new == NULL) {
|
|
|
|
pa = VM_PAGE_TO_PHYS(m_run);
|
2021-10-20 00:25:04 +00:00
|
|
|
m_new =
|
|
|
|
vm_page_alloc_noobj_contig(
|
|
|
|
req, 1, 0, pa - 1,
|
|
|
|
PAGE_SIZE, 0,
|
2015-12-19 18:42:50 +00:00
|
|
|
VM_MEMATTR_DEFAULT);
|
|
|
|
}
|
|
|
|
if (m_new == NULL) {
|
|
|
|
pa += ptoa(npages);
|
2021-10-20 00:25:04 +00:00
|
|
|
m_new =
|
|
|
|
vm_page_alloc_noobj_contig(
|
|
|
|
req, 1, pa, high, PAGE_SIZE,
|
|
|
|
0, VM_MEMATTR_DEFAULT);
|
2015-12-19 18:42:50 +00:00
|
|
|
}
|
|
|
|
if (m_new == NULL) {
|
2019-10-15 03:35:11 +00:00
|
|
|
vm_page_xunbusy(m);
|
2015-12-19 18:42:50 +00:00
|
|
|
error = ENOMEM;
|
|
|
|
goto unlock;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2019-09-16 15:09:31 +00:00
|
|
|
* Unmap the page and check for new
|
|
|
|
* wirings that may have been acquired
|
|
|
|
* through a pmap lookup.
|
2015-12-19 18:42:50 +00:00
|
|
|
*/
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
if (object->ref_count != 0 &&
|
|
|
|
!vm_page_try_remove_all(m)) {
|
2019-12-27 01:50:16 +00:00
|
|
|
vm_page_xunbusy(m);
|
2019-09-16 15:09:31 +00:00
|
|
|
vm_page_free(m_new);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
error = EBUSY;
|
|
|
|
goto unlock;
|
|
|
|
}
|
2019-09-16 15:09:31 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Replace "m" with the new page. For
|
|
|
|
* vm_page_replace(), "m" must be busy
|
|
|
|
* and dequeued. Finally, change "m"
|
|
|
|
* as if vm_page_free() was called.
|
|
|
|
*/
|
2019-12-10 18:14:50 +00:00
|
|
|
m_new->a.flags = m->a.flags &
|
2018-04-24 21:15:54 +00:00
|
|
|
~PGA_QUEUE_STATE_MASK;
|
2015-12-19 18:42:50 +00:00
|
|
|
KASSERT(m_new->oflags == VPO_UNMANAGED,
|
2018-03-23 14:38:56 +00:00
|
|
|
("page %p is managed", m_new));
|
2019-12-21 19:04:05 +00:00
|
|
|
m_new->oflags = 0;
|
2015-12-19 18:42:50 +00:00
|
|
|
pmap_copy_page(m, m_new);
|
|
|
|
m_new->valid = m->valid;
|
|
|
|
m_new->dirty = m->dirty;
|
|
|
|
m->flags &= ~PG_ZERO;
|
2018-09-06 16:17:45 +00:00
|
|
|
vm_page_dequeue(m);
|
2019-12-22 06:56:44 +00:00
|
|
|
if (vm_page_replace_hold(m_new, object,
|
|
|
|
m->pindex, m) &&
|
|
|
|
vm_page_free_prep(m))
|
2018-03-22 19:21:11 +00:00
|
|
|
SLIST_INSERT_HEAD(&free, m,
|
|
|
|
plinks.s.ss);
|
2015-12-19 18:42:50 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The new page must be deactivated
|
|
|
|
* before the object is unlocked.
|
|
|
|
*/
|
|
|
|
vm_page_deactivate(m_new);
|
|
|
|
} else {
|
|
|
|
m->flags &= ~PG_ZERO;
|
2018-09-06 16:17:45 +00:00
|
|
|
vm_page_dequeue(m);
|
2018-04-24 21:15:54 +00:00
|
|
|
if (vm_page_free_prep(m))
|
2018-03-22 19:21:11 +00:00
|
|
|
SLIST_INSERT_HEAD(&free, m,
|
|
|
|
plinks.s.ss);
|
2015-12-19 18:42:50 +00:00
|
|
|
KASSERT(m->dirty == 0,
|
|
|
|
("page %p is dirty", m));
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
error = EBUSY;
|
|
|
|
unlock:
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
} else {
|
2020-11-19 03:59:21 +00:00
|
|
|
MPASS(vm_page_domain(m) == domain);
|
2018-02-06 22:10:07 +00:00
|
|
|
vmd = VM_DOMAIN(domain);
|
|
|
|
vm_domain_free_lock(vmd);
|
2015-12-19 18:42:50 +00:00
|
|
|
order = m->order;
|
|
|
|
if (order < VM_NFREEORDER) {
|
|
|
|
/*
|
|
|
|
* The page is enqueued in the physical memory
|
2016-12-12 17:47:09 +00:00
|
|
|
* allocator's free page queues. Moreover, it
|
|
|
|
* is the first page in a power-of-two-sized
|
|
|
|
* run of contiguous free pages. Jump ahead
|
|
|
|
* to the last page within that run, and
|
|
|
|
* continue from there.
|
2015-12-19 18:42:50 +00:00
|
|
|
*/
|
|
|
|
m += (1 << order) - 1;
|
|
|
|
}
|
|
|
|
#if VM_NRESERVLEVEL > 0
|
|
|
|
else if (vm_reserv_is_page_free(m))
|
|
|
|
order = 0;
|
|
|
|
#endif
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_domain_free_unlock(vmd);
|
2015-12-19 18:42:50 +00:00
|
|
|
if (order == VM_NFREEORDER)
|
|
|
|
error = EINVAL;
|
|
|
|
}
|
|
|
|
}
|
2018-03-13 18:27:23 +00:00
|
|
|
if ((m = SLIST_FIRST(&free)) != NULL) {
|
2018-03-15 19:23:07 +00:00
|
|
|
int cnt;
|
|
|
|
|
2018-03-13 18:27:23 +00:00
|
|
|
vmd = VM_DOMAIN(domain);
|
2018-03-15 19:23:07 +00:00
|
|
|
cnt = 0;
|
2018-03-13 18:27:23 +00:00
|
|
|
vm_domain_free_lock(vmd);
|
|
|
|
do {
|
2020-11-19 03:59:21 +00:00
|
|
|
MPASS(vm_page_domain(m) == domain);
|
2018-03-13 18:27:23 +00:00
|
|
|
SLIST_REMOVE_HEAD(&free, plinks.s.ss);
|
2018-03-22 19:21:11 +00:00
|
|
|
vm_phys_free_pages(m, 0);
|
2018-03-15 19:23:07 +00:00
|
|
|
cnt++;
|
2018-03-13 18:27:23 +00:00
|
|
|
} while ((m = SLIST_FIRST(&free)) != NULL);
|
|
|
|
vm_domain_free_unlock(vmd);
|
2018-03-15 19:23:07 +00:00
|
|
|
vm_domain_freecnt_inc(vmd, cnt);
|
2018-03-13 18:27:23 +00:00
|
|
|
}
|
2015-12-19 18:42:50 +00:00
|
|
|
return (error);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define NRUNS 16
|
|
|
|
|
2023-05-08 13:25:40 +00:00
|
|
|
#define RUN_INDEX(count, nruns) ((count) % (nruns))
|
2015-12-19 18:42:50 +00:00
|
|
|
|
|
|
|
#define MIN_RECLAIM 8
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_reclaim_contig:
|
|
|
|
*
|
|
|
|
* Reclaim allocated, contiguous physical memory satisfying the specified
|
|
|
|
* conditions by relocating the virtual pages using that physical memory.
|
|
|
|
* Returns true if reclamation is successful and false otherwise. Since
|
|
|
|
* relocation requires the allocation of physical pages, reclamation may
|
2016-12-12 17:47:09 +00:00
|
|
|
* fail due to a shortage of free pages. When reclamation fails, callers
|
2018-02-20 10:13:13 +00:00
|
|
|
* are expected to perform vm_wait() before retrying a failed allocation
|
2016-12-12 17:47:09 +00:00
|
|
|
* operation, e.g., vm_page_alloc_contig().
|
2015-12-19 18:42:50 +00:00
|
|
|
*
|
|
|
|
* The caller must always specify an allocation class through "req".
|
|
|
|
*
|
|
|
|
* allocation classes:
|
|
|
|
* VM_ALLOC_NORMAL normal process request
|
|
|
|
* VM_ALLOC_SYSTEM system *really* needs a page
|
|
|
|
* VM_ALLOC_INTERRUPT interrupt time request
|
|
|
|
*
|
|
|
|
* The optional allocation flags are ignored.
|
|
|
|
*
|
|
|
|
* "npages" must be greater than zero. Both "alignment" and "boundary"
|
|
|
|
* must be a power of two.
|
|
|
|
*/
|
|
|
|
bool
|
2023-05-08 13:25:40 +00:00
|
|
|
vm_page_reclaim_contig_domain_ext(int domain, int req, u_long npages,
|
|
|
|
vm_paddr_t low, vm_paddr_t high, u_long alignment, vm_paddr_t boundary,
|
|
|
|
int desired_runs)
|
2015-12-19 18:42:50 +00:00
|
|
|
{
|
2018-02-06 22:10:07 +00:00
|
|
|
struct vm_domain *vmd;
|
2023-06-16 06:43:45 +00:00
|
|
|
vm_page_t bounds[2], m_run, _m_runs[NRUNS], *m_runs;
|
2021-03-02 15:19:53 +00:00
|
|
|
u_long count, minalign, reclaimed;
|
2023-06-16 06:43:45 +00:00
|
|
|
int error, i, min_reclaim, nruns, options, req_class, segind;
|
2023-05-08 13:25:40 +00:00
|
|
|
bool ret;
|
2015-12-19 18:42:50 +00:00
|
|
|
|
|
|
|
KASSERT(npages > 0, ("npages is 0"));
|
|
|
|
KASSERT(powerof2(alignment), ("alignment is not a power of 2"));
|
|
|
|
KASSERT(powerof2(boundary), ("boundary is not a power of 2"));
|
2021-03-02 15:19:53 +00:00
|
|
|
|
2023-05-08 13:25:40 +00:00
|
|
|
ret = false;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the caller wants to reclaim multiple runs, try to allocate
|
|
|
|
* space to store the runs. If that fails, fall back to the old
|
|
|
|
* behavior of just reclaiming MIN_RECLAIM pages.
|
|
|
|
*/
|
|
|
|
if (desired_runs > 1)
|
|
|
|
m_runs = malloc((NRUNS + desired_runs) * sizeof(*m_runs),
|
|
|
|
M_TEMP, M_NOWAIT);
|
|
|
|
else
|
|
|
|
m_runs = NULL;
|
|
|
|
|
|
|
|
if (m_runs == NULL) {
|
|
|
|
m_runs = _m_runs;
|
|
|
|
nruns = NRUNS;
|
|
|
|
} else {
|
|
|
|
nruns = NRUNS + desired_runs - 1;
|
|
|
|
}
|
|
|
|
min_reclaim = MAX(desired_runs * npages, MIN_RECLAIM);
|
|
|
|
|
2021-03-02 15:19:53 +00:00
|
|
|
/*
|
|
|
|
* The caller will attempt an allocation after some runs have been
|
|
|
|
* reclaimed and added to the vm_phys buddy lists. Due to limitations
|
|
|
|
* of vm_phys_alloc_contig(), round up the requested length to the next
|
|
|
|
* power of two or maximum chunk size, and ensure that each run is
|
|
|
|
* suitably aligned.
|
|
|
|
*/
|
|
|
|
minalign = 1ul << imin(flsl(npages - 1), VM_NFREEORDER - 1);
|
|
|
|
npages = roundup2(npages, minalign);
|
|
|
|
if (alignment < ptoa(minalign))
|
|
|
|
alignment = ptoa(minalign);
|
2015-12-19 18:42:50 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The page daemon is allowed to dig deeper into the free page list.
|
|
|
|
*/
|
2021-03-02 15:19:53 +00:00
|
|
|
req_class = req & VM_ALLOC_CLASS_MASK;
|
2015-12-19 18:42:50 +00:00
|
|
|
if (curproc == pageproc && req_class != VM_ALLOC_INTERRUPT)
|
|
|
|
req_class = VM_ALLOC_SYSTEM;
|
|
|
|
|
|
|
|
/*
|
2016-12-12 17:47:09 +00:00
|
|
|
* Return if the number of free pages cannot satisfy the requested
|
|
|
|
* allocation.
|
2015-12-19 18:42:50 +00:00
|
|
|
*/
|
2018-02-06 22:10:07 +00:00
|
|
|
vmd = VM_DOMAIN(domain);
|
|
|
|
count = vmd->vmd_free_count;
|
|
|
|
if (count < npages + vmd->vmd_free_reserved || (count < npages +
|
|
|
|
vmd->vmd_interrupt_free_min && req_class == VM_ALLOC_SYSTEM) ||
|
2015-12-19 18:42:50 +00:00
|
|
|
(count < npages && req_class == VM_ALLOC_INTERRUPT))
|
2023-05-08 13:25:40 +00:00
|
|
|
goto done;
|
2015-12-19 18:42:50 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Scan up to three times, relaxing the restrictions ("options") on
|
|
|
|
* the reclamation of reservations and superpages each time.
|
|
|
|
*/
|
|
|
|
for (options = VPSC_NORESERV;;) {
|
|
|
|
/*
|
|
|
|
* Find the highest runs that satisfy the given constraints
|
|
|
|
* and restrictions, and record them in "m_runs".
|
|
|
|
*/
|
|
|
|
count = 0;
|
2023-06-16 06:43:45 +00:00
|
|
|
segind = vm_phys_lookup_segind(low);
|
|
|
|
while ((segind = vm_phys_find_range(bounds, segind, domain,
|
|
|
|
npages, low, high)) != -1) {
|
|
|
|
while ((m_run = vm_page_scan_contig(npages, bounds[0],
|
|
|
|
bounds[1], alignment, boundary, options))) {
|
|
|
|
bounds[0] = m_run + npages;
|
|
|
|
m_runs[RUN_INDEX(count, nruns)] = m_run;
|
|
|
|
count++;
|
|
|
|
}
|
|
|
|
segind++;
|
2015-12-19 18:42:50 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reclaim the highest runs in LIFO (descending) order until
|
|
|
|
* the number of reclaimed pages, "reclaimed", is at least
|
2023-05-08 13:25:40 +00:00
|
|
|
* "min_reclaim". Reset "reclaimed" each time because each
|
2015-12-19 18:42:50 +00:00
|
|
|
* reclamation is idempotent, and runs will (likely) recur
|
|
|
|
* from one scan to the next as restrictions are relaxed.
|
|
|
|
*/
|
|
|
|
reclaimed = 0;
|
2023-05-08 13:25:40 +00:00
|
|
|
for (i = 0; count > 0 && i < nruns; i++) {
|
2015-12-19 18:42:50 +00:00
|
|
|
count--;
|
2023-05-08 13:25:40 +00:00
|
|
|
m_run = m_runs[RUN_INDEX(count, nruns)];
|
2018-02-06 22:10:07 +00:00
|
|
|
error = vm_page_reclaim_run(req_class, domain, npages,
|
|
|
|
m_run, high);
|
2015-12-19 18:42:50 +00:00
|
|
|
if (error == 0) {
|
|
|
|
reclaimed += npages;
|
2023-05-08 13:25:40 +00:00
|
|
|
if (reclaimed >= min_reclaim) {
|
|
|
|
ret = true;
|
|
|
|
goto done;
|
|
|
|
}
|
2015-12-19 18:42:50 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Either relax the restrictions on the next scan or return if
|
|
|
|
* the last scan had no restrictions.
|
|
|
|
*/
|
|
|
|
if (options == VPSC_NORESERV)
|
|
|
|
options = VPSC_NOSUPER;
|
|
|
|
else if (options == VPSC_NOSUPER)
|
|
|
|
options = VPSC_ANY;
|
2023-05-08 13:25:40 +00:00
|
|
|
else if (options == VPSC_ANY) {
|
|
|
|
ret = reclaimed != 0;
|
|
|
|
goto done;
|
|
|
|
}
|
2015-12-19 18:42:50 +00:00
|
|
|
}
|
2023-05-08 13:25:40 +00:00
|
|
|
done:
|
|
|
|
if (m_runs != _m_runs)
|
|
|
|
free(m_runs, M_TEMP);
|
|
|
|
return (ret);
|
|
|
|
}
|
|
|
|
|
|
|
|
bool
|
|
|
|
vm_page_reclaim_contig_domain(int domain, int req, u_long npages,
|
|
|
|
vm_paddr_t low, vm_paddr_t high, u_long alignment, vm_paddr_t boundary)
|
|
|
|
{
|
|
|
|
return (vm_page_reclaim_contig_domain_ext(domain, req, npages, low, high,
|
|
|
|
alignment, boundary, 1));
|
2015-12-19 18:42:50 +00:00
|
|
|
}
|
|
|
|
|
2018-01-12 22:48:23 +00:00
|
|
|
bool
|
|
|
|
vm_page_reclaim_contig(int req, u_long npages, vm_paddr_t low, vm_paddr_t high,
|
|
|
|
u_long alignment, vm_paddr_t boundary)
|
|
|
|
{
|
|
|
|
struct vm_domainset_iter di;
|
|
|
|
int domain;
|
|
|
|
bool ret;
|
|
|
|
|
2018-03-29 02:54:50 +00:00
|
|
|
vm_domainset_iter_page_init(&di, NULL, 0, &domain, &req);
|
2018-01-12 22:48:23 +00:00
|
|
|
do {
|
|
|
|
ret = vm_page_reclaim_contig_domain(domain, req, npages, low,
|
|
|
|
high, alignment, boundary);
|
|
|
|
if (ret)
|
|
|
|
break;
|
2018-10-23 16:35:58 +00:00
|
|
|
} while (vm_domainset_iter_page(&di, NULL, &domain) == 0);
|
2018-01-12 22:48:23 +00:00
|
|
|
|
|
|
|
return (ret);
|
|
|
|
}
|
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
/*
|
|
|
|
* Set the domain in the appropriate page level domainset.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_domain_set(struct vm_domain *vmd)
|
|
|
|
{
|
|
|
|
|
|
|
|
mtx_lock(&vm_domainset_lock);
|
|
|
|
if (!vmd->vmd_minset && vm_paging_min(vmd)) {
|
|
|
|
vmd->vmd_minset = 1;
|
|
|
|
DOMAINSET_SET(vmd->vmd_domain, &vm_min_domains);
|
|
|
|
}
|
|
|
|
if (!vmd->vmd_severeset && vm_paging_severe(vmd)) {
|
|
|
|
vmd->vmd_severeset = 1;
|
2018-06-07 13:29:54 +00:00
|
|
|
DOMAINSET_SET(vmd->vmd_domain, &vm_severe_domains);
|
2018-02-06 22:10:07 +00:00
|
|
|
}
|
|
|
|
mtx_unlock(&vm_domainset_lock);
|
|
|
|
}
|
2018-01-12 22:48:23 +00:00
|
|
|
|
1998-12-23 01:52:47 +00:00
|
|
|
/*
|
2018-02-06 22:10:07 +00:00
|
|
|
* Clear the domain from the appropriate page level domainset.
|
|
|
|
*/
|
2018-03-15 19:23:07 +00:00
|
|
|
void
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_domain_clear(struct vm_domain *vmd)
|
|
|
|
{
|
|
|
|
|
|
|
|
mtx_lock(&vm_domainset_lock);
|
|
|
|
if (vmd->vmd_minset && !vm_paging_min(vmd)) {
|
|
|
|
vmd->vmd_minset = 0;
|
|
|
|
DOMAINSET_CLR(vmd->vmd_domain, &vm_min_domains);
|
|
|
|
if (vm_min_waiters != 0) {
|
|
|
|
vm_min_waiters = 0;
|
|
|
|
wakeup(&vm_min_domains);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (vmd->vmd_severeset && !vm_paging_severe(vmd)) {
|
|
|
|
vmd->vmd_severeset = 0;
|
|
|
|
DOMAINSET_CLR(vmd->vmd_domain, &vm_severe_domains);
|
|
|
|
if (vm_severe_waiters != 0) {
|
|
|
|
vm_severe_waiters = 0;
|
|
|
|
wakeup(&vm_severe_domains);
|
|
|
|
}
|
|
|
|
}
|
2018-03-15 19:23:07 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If pageout daemon needs pages, then tell it that there are
|
|
|
|
* some free.
|
|
|
|
*/
|
|
|
|
if (vmd->vmd_pageout_pages_needed &&
|
|
|
|
vmd->vmd_free_count >= vmd->vmd_pageout_free_min) {
|
|
|
|
wakeup(&vmd->vmd_pageout_pages_needed);
|
|
|
|
vmd->vmd_pageout_pages_needed = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* See comments in vm_wait_doms(). */
|
|
|
|
if (vm_pageproc_waiters) {
|
|
|
|
vm_pageproc_waiters = 0;
|
|
|
|
wakeup(&vm_pageproc_waiters);
|
|
|
|
}
|
2018-02-06 22:10:07 +00:00
|
|
|
mtx_unlock(&vm_domainset_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Wait for free pages to exceed the min threshold globally.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_wait_min(void)
|
|
|
|
{
|
|
|
|
|
|
|
|
mtx_lock(&vm_domainset_lock);
|
|
|
|
while (vm_page_count_min()) {
|
|
|
|
vm_min_waiters++;
|
|
|
|
msleep(&vm_min_domains, &vm_domainset_lock, PVM, "vmwait", 0);
|
|
|
|
}
|
|
|
|
mtx_unlock(&vm_domainset_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Wait for free pages to exceed the severe threshold globally.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_wait_severe(void)
|
|
|
|
{
|
|
|
|
|
|
|
|
mtx_lock(&vm_domainset_lock);
|
|
|
|
while (vm_page_count_severe()) {
|
|
|
|
vm_severe_waiters++;
|
|
|
|
msleep(&vm_severe_domains, &vm_domainset_lock, PVM,
|
|
|
|
"vmwait", 0);
|
|
|
|
}
|
|
|
|
mtx_unlock(&vm_domainset_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
u_int
|
|
|
|
vm_wait_count(void)
|
|
|
|
{
|
|
|
|
|
2018-03-15 19:23:07 +00:00
|
|
|
return (vm_severe_waiters + vm_min_waiters + vm_pageproc_waiters);
|
2018-02-20 10:13:13 +00:00
|
|
|
}
|
|
|
|
|
2020-09-08 23:28:09 +00:00
|
|
|
int
|
|
|
|
vm_wait_doms(const domainset_t *wdoms, int mflags)
|
2018-02-20 10:13:13 +00:00
|
|
|
{
|
2020-09-08 23:28:09 +00:00
|
|
|
int error;
|
|
|
|
|
|
|
|
error = 0;
|
2018-02-06 22:10:07 +00:00
|
|
|
|
2018-02-20 10:13:13 +00:00
|
|
|
/*
|
|
|
|
* We use racey wakeup synchronization to avoid expensive global
|
|
|
|
* locking for the pageproc when sleeping with a non-specific vm_wait.
|
|
|
|
* To handle this, we only sleep for one tick in this instance. It
|
|
|
|
* is expected that most allocations for the pageproc will come from
|
|
|
|
* kmem or vm_page_grab* which will use the more specific and
|
|
|
|
* race-free vm_wait_domain().
|
|
|
|
*/
|
|
|
|
if (curproc == pageproc) {
|
|
|
|
mtx_lock(&vm_domainset_lock);
|
|
|
|
vm_pageproc_waiters++;
|
2020-09-08 23:28:09 +00:00
|
|
|
error = msleep(&vm_pageproc_waiters, &vm_domainset_lock,
|
|
|
|
PVM | PDROP | mflags, "pageprocwait", 1);
|
2018-02-20 10:13:13 +00:00
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* XXX Ideally we would wait only until the allocation could
|
|
|
|
* be satisfied. This condition can cause new allocators to
|
|
|
|
* consume all freed pages while old allocators wait.
|
|
|
|
*/
|
|
|
|
mtx_lock(&vm_domainset_lock);
|
2018-09-24 19:24:17 +00:00
|
|
|
if (vm_page_count_min_set(wdoms)) {
|
2022-04-14 19:45:54 +00:00
|
|
|
if (pageproc == NULL)
|
|
|
|
panic("vm_wait in early boot");
|
2018-02-20 10:13:13 +00:00
|
|
|
vm_min_waiters++;
|
2020-09-08 23:28:09 +00:00
|
|
|
error = msleep(&vm_min_domains, &vm_domainset_lock,
|
|
|
|
PVM | PDROP | mflags, "vmwait", 0);
|
2018-09-06 19:28:52 +00:00
|
|
|
} else
|
|
|
|
mtx_unlock(&vm_domainset_lock);
|
2018-02-20 10:13:13 +00:00
|
|
|
}
|
2020-09-08 23:28:09 +00:00
|
|
|
return (error);
|
2018-02-06 22:10:07 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_wait_domain:
|
1998-12-23 01:52:47 +00:00
|
|
|
*
|
2012-10-03 05:06:45 +00:00
|
|
|
* Sleep until free pages are available for allocation.
|
2018-02-06 22:10:07 +00:00
|
|
|
* - Called in various places after failed memory allocations.
|
1998-12-23 01:52:47 +00:00
|
|
|
*/
|
2018-02-06 22:10:07 +00:00
|
|
|
void
|
|
|
|
vm_wait_domain(int domain)
|
1996-11-28 23:15:07 +00:00
|
|
|
{
|
2018-02-06 22:10:07 +00:00
|
|
|
struct vm_domain *vmd;
|
2018-02-20 10:13:13 +00:00
|
|
|
domainset_t wdom;
|
2018-02-06 22:10:07 +00:00
|
|
|
|
|
|
|
vmd = VM_DOMAIN(domain);
|
2018-03-15 19:23:07 +00:00
|
|
|
vm_domain_free_assert_unlocked(vmd);
|
1996-11-28 23:15:07 +00:00
|
|
|
|
|
|
|
if (curproc == pageproc) {
|
2018-03-15 19:23:07 +00:00
|
|
|
mtx_lock(&vm_domainset_lock);
|
|
|
|
if (vmd->vmd_free_count < vmd->vmd_pageout_free_min) {
|
|
|
|
vmd->vmd_pageout_pages_needed = 1;
|
|
|
|
msleep(&vmd->vmd_pageout_pages_needed,
|
|
|
|
&vm_domainset_lock, PDROP | PSWP, "VMWait", 0);
|
|
|
|
} else
|
|
|
|
mtx_unlock(&vm_domainset_lock);
|
1996-11-28 23:15:07 +00:00
|
|
|
} else {
|
2018-02-20 10:13:13 +00:00
|
|
|
DOMAINSET_ZERO(&wdom);
|
|
|
|
DOMAINSET_SET(vmd->vmd_domain, &wdom);
|
2020-09-08 23:28:09 +00:00
|
|
|
vm_wait_doms(&wdom, 0);
|
1996-11-28 23:15:07 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-09-08 23:28:09 +00:00
|
|
|
static int
|
|
|
|
vm_wait_flags(vm_object_t obj, int mflags)
|
2017-11-08 02:39:37 +00:00
|
|
|
{
|
2018-02-20 10:13:13 +00:00
|
|
|
struct domainset *d;
|
|
|
|
|
|
|
|
d = NULL;
|
2017-11-08 02:39:37 +00:00
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
/*
|
2018-02-20 10:13:13 +00:00
|
|
|
* Carefully fetch pointers only once: the struct domainset
|
|
|
|
* itself is ummutable but the pointer might change.
|
2018-02-06 22:10:07 +00:00
|
|
|
*/
|
2018-02-20 10:13:13 +00:00
|
|
|
if (obj != NULL)
|
|
|
|
d = obj->domain.dr_policy;
|
|
|
|
if (d == NULL)
|
|
|
|
d = curthread->td_domain.dr_policy;
|
|
|
|
|
2020-09-08 23:28:09 +00:00
|
|
|
return (vm_wait_doms(&d->ds_mask, mflags));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_wait:
|
|
|
|
*
|
|
|
|
* Sleep until free pages are available for allocation in the
|
|
|
|
* affinity domains of the obj. If obj is NULL, the domain set
|
|
|
|
* for the calling thread is used.
|
|
|
|
* Called in various places after failed memory allocations.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_wait(vm_object_t obj)
|
|
|
|
{
|
|
|
|
(void)vm_wait_flags(obj, 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
vm_wait_intr(vm_object_t obj)
|
|
|
|
{
|
|
|
|
return (vm_wait_flags(obj, PCATCH));
|
2017-11-08 02:39:37 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2018-02-06 22:10:07 +00:00
|
|
|
* vm_domain_alloc_fail:
|
2017-11-08 02:39:37 +00:00
|
|
|
*
|
|
|
|
* Called when a page allocation function fails. Informs the
|
|
|
|
* pagedaemon and performs the requested wait. Requires the
|
2018-02-06 22:10:07 +00:00
|
|
|
* domain_free and object lock on entry. Returns with the
|
2017-11-08 02:39:37 +00:00
|
|
|
* object lock held and free lock released. Returns an error when
|
|
|
|
* retry is necessary.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
static int
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_domain_alloc_fail(struct vm_domain *vmd, vm_object_t object, int req)
|
2017-11-08 02:39:37 +00:00
|
|
|
{
|
|
|
|
|
2018-03-15 19:23:07 +00:00
|
|
|
vm_domain_free_assert_unlocked(vmd);
|
2017-11-08 02:39:37 +00:00
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
atomic_add_int(&vmd->vmd_pageout_deficit,
|
2017-11-08 02:39:37 +00:00
|
|
|
max((u_int)req >> VM_ALLOC_COUNT_SHIFT, 1));
|
|
|
|
if (req & (VM_ALLOC_WAITOK | VM_ALLOC_WAITFAIL)) {
|
|
|
|
if (object != NULL)
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_wait_domain(vmd->vmd_domain);
|
2017-11-08 02:39:37 +00:00
|
|
|
if (object != NULL)
|
|
|
|
VM_OBJECT_WLOCK(object);
|
|
|
|
if (req & VM_ALLOC_WAITOK)
|
|
|
|
return (EAGAIN);
|
2017-12-24 19:45:16 +00:00
|
|
|
}
|
2018-03-15 19:23:07 +00:00
|
|
|
|
2017-11-08 02:39:37 +00:00
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
|
2002-02-19 18:34:02 +00:00
|
|
|
/*
|
2018-02-20 10:13:13 +00:00
|
|
|
* vm_waitpfault:
|
2002-02-19 18:34:02 +00:00
|
|
|
*
|
2012-10-03 05:06:45 +00:00
|
|
|
* Sleep until free pages are available for allocation.
|
2002-02-19 18:34:02 +00:00
|
|
|
* - Called only in vm_fault so that processes page faulting
|
|
|
|
* can be easily tracked.
|
2002-02-19 18:50:49 +00:00
|
|
|
* - Sleeps at a lower priority than vm_wait() so that vm_wait()ing
|
|
|
|
* processes will be able to grab memory first. Do not change
|
|
|
|
* this balance without careful testing first.
|
2002-02-19 18:34:02 +00:00
|
|
|
*/
|
|
|
|
void
|
2019-08-16 09:43:49 +00:00
|
|
|
vm_waitpfault(struct domainset *dset, int timo)
|
2002-02-19 18:34:02 +00:00
|
|
|
{
|
|
|
|
|
2018-09-06 19:28:52 +00:00
|
|
|
/*
|
|
|
|
* XXX Ideally we would wait only until the allocation could
|
|
|
|
* be satisfied. This condition can cause new allocators to
|
|
|
|
* consume all freed pages while old allocators wait.
|
|
|
|
*/
|
2018-02-06 22:10:07 +00:00
|
|
|
mtx_lock(&vm_domainset_lock);
|
2018-09-24 19:24:17 +00:00
|
|
|
if (vm_page_count_min_set(&dset->ds_mask)) {
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_min_waiters++;
|
2018-09-06 19:28:52 +00:00
|
|
|
msleep(&vm_min_domains, &vm_domainset_lock, PUSER | PDROP,
|
2019-08-16 09:43:49 +00:00
|
|
|
"pfault", timo);
|
2018-09-06 19:28:52 +00:00
|
|
|
} else
|
|
|
|
mtx_unlock(&vm_domainset_lock);
|
2002-02-19 18:34:02 +00:00
|
|
|
}
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
static struct vm_pagequeue *
|
|
|
|
_vm_page_pagequeue(vm_page_t m, uint8_t queue)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (&vm_pagequeue_domain(m)->vmd_pagequeues[queue]);
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef INVARIANTS
|
2019-09-16 15:04:45 +00:00
|
|
|
static struct vm_pagequeue *
|
|
|
|
vm_page_pagequeue(vm_page_t m)
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
{
|
2019-09-16 15:03:12 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
return (_vm_page_pagequeue(m, vm_page_astate_load(m).queue));
|
|
|
|
}
|
|
|
|
#endif
|
2019-09-16 15:03:12 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
static __always_inline bool
|
|
|
|
vm_page_pqstate_fcmpset(vm_page_t m, vm_page_astate_t *old, vm_page_astate_t new)
|
|
|
|
{
|
|
|
|
vm_page_astate_t tmp;
|
|
|
|
|
|
|
|
tmp = *old;
|
|
|
|
do {
|
|
|
|
if (__predict_true(vm_page_astate_fcmpset(m, old, new)))
|
|
|
|
return (true);
|
|
|
|
counter_u64_add(pqstate_commit_retries, 1);
|
|
|
|
} while (old->_bits == tmp._bits);
|
|
|
|
|
|
|
|
return (false);
|
2018-04-24 21:15:54 +00:00
|
|
|
}
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
/*
|
|
|
|
* Do the work of committing a queue state update that moves the page out of
|
|
|
|
* its current queue.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
_vm_page_pqstate_commit_dequeue(struct vm_pagequeue *pq, vm_page_t m,
|
|
|
|
vm_page_astate_t *old, vm_page_astate_t new)
|
2018-04-24 21:15:54 +00:00
|
|
|
{
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_t next;
|
2018-04-24 21:15:54 +00:00
|
|
|
|
|
|
|
vm_pagequeue_assert_locked(pq);
|
2019-12-28 19:03:32 +00:00
|
|
|
KASSERT(vm_page_pagequeue(m) == pq,
|
|
|
|
("%s: queue %p does not match page %p", __func__, pq, m));
|
|
|
|
KASSERT(old->queue != PQ_NONE && new.queue != old->queue,
|
|
|
|
("%s: invalid queue indices %d %d",
|
|
|
|
__func__, old->queue, new.queue));
|
2018-04-24 21:15:54 +00:00
|
|
|
|
2018-09-08 21:49:43 +00:00
|
|
|
/*
|
2019-12-28 19:03:32 +00:00
|
|
|
* Once the queue index of the page changes there is nothing
|
|
|
|
* synchronizing with further updates to the page's physical
|
|
|
|
* queue state. Therefore we must speculatively remove the page
|
|
|
|
* from the queue now and be prepared to roll back if the queue
|
|
|
|
* state update fails. If the page is not physically enqueued then
|
|
|
|
* we just update its queue index.
|
2018-09-08 21:49:43 +00:00
|
|
|
*/
|
2019-12-28 19:03:32 +00:00
|
|
|
if ((old->flags & PGA_ENQUEUED) != 0) {
|
|
|
|
new.flags &= ~PGA_ENQUEUED;
|
|
|
|
next = TAILQ_NEXT(m, plinks.q);
|
|
|
|
TAILQ_REMOVE(&pq->pq_pl, m, plinks.q);
|
|
|
|
vm_pagequeue_cnt_dec(pq);
|
|
|
|
if (!vm_page_pqstate_fcmpset(m, old, new)) {
|
|
|
|
if (next == NULL)
|
|
|
|
TAILQ_INSERT_TAIL(&pq->pq_pl, m, plinks.q);
|
|
|
|
else
|
|
|
|
TAILQ_INSERT_BEFORE(next, m, plinks.q);
|
2019-09-16 15:03:12 +00:00
|
|
|
vm_pagequeue_cnt_inc(pq);
|
2019-12-28 19:03:32 +00:00
|
|
|
return (false);
|
|
|
|
} else {
|
|
|
|
return (true);
|
2019-09-16 15:04:45 +00:00
|
|
|
}
|
2019-12-28 19:03:32 +00:00
|
|
|
} else {
|
|
|
|
return (vm_page_pqstate_fcmpset(m, old, new));
|
|
|
|
}
|
|
|
|
}
|
2019-08-26 20:20:10 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
static bool
|
|
|
|
vm_page_pqstate_commit_dequeue(vm_page_t m, vm_page_astate_t *old,
|
|
|
|
vm_page_astate_t new)
|
|
|
|
{
|
|
|
|
struct vm_pagequeue *pq;
|
|
|
|
vm_page_astate_t as;
|
|
|
|
bool ret;
|
2019-09-16 15:04:45 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
pq = _vm_page_pagequeue(m, old->queue);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The queue field and PGA_ENQUEUED flag are stable only so long as the
|
|
|
|
* corresponding page queue lock is held.
|
|
|
|
*/
|
|
|
|
vm_pagequeue_lock(pq);
|
|
|
|
as = vm_page_astate_load(m);
|
|
|
|
if (__predict_false(as._bits != old->_bits)) {
|
|
|
|
*old = as;
|
|
|
|
ret = false;
|
2019-09-25 17:08:35 +00:00
|
|
|
} else {
|
2019-12-28 19:03:32 +00:00
|
|
|
ret = _vm_page_pqstate_commit_dequeue(pq, m, old, new);
|
|
|
|
}
|
|
|
|
vm_pagequeue_unlock(pq);
|
|
|
|
return (ret);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Commit a queue state update that enqueues or requeues a page.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
_vm_page_pqstate_commit_requeue(struct vm_pagequeue *pq, vm_page_t m,
|
|
|
|
vm_page_astate_t *old, vm_page_astate_t new)
|
|
|
|
{
|
|
|
|
struct vm_domain *vmd;
|
|
|
|
|
|
|
|
vm_pagequeue_assert_locked(pq);
|
|
|
|
KASSERT(old->queue != PQ_NONE && new.queue == old->queue,
|
|
|
|
("%s: invalid queue indices %d %d",
|
|
|
|
__func__, old->queue, new.queue));
|
|
|
|
|
|
|
|
new.flags |= PGA_ENQUEUED;
|
|
|
|
if (!vm_page_pqstate_fcmpset(m, old, new))
|
|
|
|
return (false);
|
|
|
|
|
|
|
|
if ((old->flags & PGA_ENQUEUED) != 0)
|
|
|
|
TAILQ_REMOVE(&pq->pq_pl, m, plinks.q);
|
|
|
|
else
|
|
|
|
vm_pagequeue_cnt_inc(pq);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Give PGA_REQUEUE_HEAD precedence over PGA_REQUEUE. In particular, if
|
|
|
|
* both flags are set in close succession, only PGA_REQUEUE_HEAD will be
|
|
|
|
* applied, even if it was set first.
|
|
|
|
*/
|
|
|
|
if ((old->flags & PGA_REQUEUE_HEAD) != 0) {
|
|
|
|
vmd = vm_pagequeue_domain(m);
|
|
|
|
KASSERT(pq == &vmd->vmd_pagequeues[PQ_INACTIVE],
|
|
|
|
("%s: invalid page queue for page %p", __func__, m));
|
|
|
|
TAILQ_INSERT_BEFORE(&vmd->vmd_inacthead, m, plinks.q);
|
|
|
|
} else {
|
|
|
|
TAILQ_INSERT_TAIL(&pq->pq_pl, m, plinks.q);
|
|
|
|
}
|
|
|
|
return (true);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Commit a queue state update that encodes a request for a deferred queue
|
|
|
|
* operation.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
vm_page_pqstate_commit_request(vm_page_t m, vm_page_astate_t *old,
|
|
|
|
vm_page_astate_t new)
|
|
|
|
{
|
|
|
|
|
|
|
|
KASSERT(old->queue == new.queue || new.queue != PQ_NONE,
|
|
|
|
("%s: invalid state, queue %d flags %x",
|
|
|
|
__func__, new.queue, new.flags));
|
|
|
|
|
|
|
|
if (old->_bits != new._bits &&
|
|
|
|
!vm_page_pqstate_fcmpset(m, old, new))
|
|
|
|
return (false);
|
|
|
|
vm_page_pqbatch_submit(m, new.queue);
|
|
|
|
return (true);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* A generic queue state update function. This handles more cases than the
|
|
|
|
* specialized functions above.
|
|
|
|
*/
|
|
|
|
bool
|
|
|
|
vm_page_pqstate_commit(vm_page_t m, vm_page_astate_t *old, vm_page_astate_t new)
|
|
|
|
{
|
|
|
|
|
|
|
|
if (old->_bits == new._bits)
|
|
|
|
return (true);
|
|
|
|
|
|
|
|
if (old->queue != PQ_NONE && new.queue != old->queue) {
|
|
|
|
if (!vm_page_pqstate_commit_dequeue(m, old, new))
|
|
|
|
return (false);
|
|
|
|
if (new.queue != PQ_NONE)
|
|
|
|
vm_page_pqbatch_submit(m, new.queue);
|
|
|
|
} else {
|
|
|
|
if (!vm_page_pqstate_fcmpset(m, old, new))
|
|
|
|
return (false);
|
|
|
|
if (new.queue != PQ_NONE &&
|
|
|
|
((new.flags & ~old->flags) & PGA_QUEUE_OP_MASK) != 0)
|
|
|
|
vm_page_pqbatch_submit(m, new.queue);
|
|
|
|
}
|
|
|
|
return (true);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Apply deferred queue state updates to a page.
|
|
|
|
*/
|
|
|
|
static inline void
|
|
|
|
vm_pqbatch_process_page(struct vm_pagequeue *pq, vm_page_t m, uint8_t queue)
|
|
|
|
{
|
|
|
|
vm_page_astate_t new, old;
|
|
|
|
|
|
|
|
CRITICAL_ASSERT(curthread);
|
|
|
|
vm_pagequeue_assert_locked(pq);
|
|
|
|
KASSERT(queue < PQ_COUNT,
|
|
|
|
("%s: invalid queue index %d", __func__, queue));
|
|
|
|
KASSERT(pq == _vm_page_pagequeue(m, queue),
|
|
|
|
("%s: page %p does not belong to queue %p", __func__, m, pq));
|
|
|
|
|
|
|
|
for (old = vm_page_astate_load(m);;) {
|
|
|
|
if (__predict_false(old.queue != queue ||
|
|
|
|
(old.flags & PGA_QUEUE_OP_MASK) == 0)) {
|
|
|
|
counter_u64_add(queue_nops, 1);
|
|
|
|
break;
|
|
|
|
}
|
2021-02-24 23:56:16 +00:00
|
|
|
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
|
|
|
|
("%s: page %p is unmanaged", __func__, m));
|
2019-12-28 19:03:32 +00:00
|
|
|
|
|
|
|
new = old;
|
|
|
|
if ((old.flags & PGA_DEQUEUE) != 0) {
|
|
|
|
new.flags &= ~PGA_QUEUE_OP_MASK;
|
|
|
|
new.queue = PQ_NONE;
|
|
|
|
if (__predict_true(_vm_page_pqstate_commit_dequeue(pq,
|
|
|
|
m, &old, new))) {
|
|
|
|
counter_u64_add(queue_ops, 1);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
new.flags &= ~(PGA_REQUEUE | PGA_REQUEUE_HEAD);
|
|
|
|
if (__predict_true(_vm_page_pqstate_commit_requeue(pq,
|
|
|
|
m, &old, new))) {
|
|
|
|
counter_u64_add(queue_ops, 1);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2018-04-24 21:15:54 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
|
|
|
vm_pqbatch_process(struct vm_pagequeue *pq, struct vm_batchqueue *bq,
|
|
|
|
uint8_t queue)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
for (i = 0; i < bq->bq_cnt; i++)
|
|
|
|
vm_pqbatch_process_page(pq, bq->bq_pa[i], queue);
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_batchqueue_init(bq);
|
|
|
|
}
|
|
|
|
|
2019-09-03 14:29:58 +00:00
|
|
|
/*
|
|
|
|
* vm_page_pqbatch_submit: [ internal use only ]
|
|
|
|
*
|
|
|
|
* Enqueue a page in the specified page queue's batched work queue.
|
2019-09-16 15:04:45 +00:00
|
|
|
* The caller must have encoded the requested operation in the page
|
2019-12-10 18:14:50 +00:00
|
|
|
* structure's a.flags field.
|
2019-09-03 14:29:58 +00:00
|
|
|
*/
|
2019-08-23 19:49:29 +00:00
|
|
|
void
|
|
|
|
vm_page_pqbatch_submit(vm_page_t m, uint8_t queue)
|
2018-04-24 21:15:54 +00:00
|
|
|
{
|
|
|
|
struct vm_batchqueue *bq;
|
|
|
|
struct vm_pagequeue *pq;
|
2022-12-14 19:34:07 +00:00
|
|
|
int domain, slots_remaining;
|
2018-04-24 21:15:54 +00:00
|
|
|
|
|
|
|
KASSERT(queue < PQ_COUNT, ("invalid queue %d", queue));
|
|
|
|
|
2020-11-19 03:59:21 +00:00
|
|
|
domain = vm_page_domain(m);
|
2018-04-24 21:15:54 +00:00
|
|
|
critical_enter();
|
|
|
|
bq = DPCPU_PTR(pqbatch[domain][queue]);
|
2022-12-14 19:34:07 +00:00
|
|
|
slots_remaining = vm_batchqueue_insert(bq, m);
|
|
|
|
if (slots_remaining > (VM_BATCHQUEUE_SIZE >> 1)) {
|
|
|
|
/* keep building the bq */
|
|
|
|
critical_exit();
|
|
|
|
return;
|
|
|
|
} else if (slots_remaining > 0 ) {
|
|
|
|
/* Try to process the bq if we can get the lock */
|
|
|
|
pq = &VM_DOMAIN(domain)->vmd_pagequeues[queue];
|
|
|
|
if (vm_pagequeue_trylock(pq)) {
|
|
|
|
vm_pqbatch_process(pq, bq, queue);
|
|
|
|
vm_pagequeue_unlock(pq);
|
|
|
|
}
|
2018-04-24 21:15:54 +00:00
|
|
|
critical_exit();
|
|
|
|
return;
|
|
|
|
}
|
2019-10-07 23:19:09 +00:00
|
|
|
critical_exit();
|
2020-11-19 15:40:58 +00:00
|
|
|
|
2022-12-14 19:34:07 +00:00
|
|
|
/* if we make it here, the bq is full so wait for the lock */
|
|
|
|
|
2020-11-19 15:40:58 +00:00
|
|
|
pq = &VM_DOMAIN(domain)->vmd_pagequeues[queue];
|
2019-10-07 23:19:09 +00:00
|
|
|
vm_pagequeue_lock(pq);
|
|
|
|
critical_enter();
|
|
|
|
bq = DPCPU_PTR(pqbatch[domain][queue]);
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_pqbatch_process(pq, bq, queue);
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_pqbatch_process_page(pq, m, queue);
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_pagequeue_unlock(pq);
|
|
|
|
critical_exit();
|
|
|
|
}
|
|
|
|
|
2010-05-09 16:55:42 +00:00
|
|
|
/*
|
2019-08-23 19:49:29 +00:00
|
|
|
* vm_page_pqbatch_drain: [ internal use only ]
|
2010-05-09 16:55:42 +00:00
|
|
|
*
|
2018-04-24 21:15:54 +00:00
|
|
|
* Force all per-CPU page queue batch queues to be drained. This is
|
|
|
|
* intended for use in severe memory shortages, to ensure that pages
|
|
|
|
* do not remain stuck in the batch queues.
|
|
|
|
*/
|
|
|
|
void
|
2019-08-23 19:49:29 +00:00
|
|
|
vm_page_pqbatch_drain(void)
|
2018-04-24 21:15:54 +00:00
|
|
|
{
|
|
|
|
struct thread *td;
|
|
|
|
struct vm_domain *vmd;
|
|
|
|
struct vm_pagequeue *pq;
|
|
|
|
int cpu, domain, queue;
|
|
|
|
|
|
|
|
td = curthread;
|
|
|
|
CPU_FOREACH(cpu) {
|
|
|
|
thread_lock(td);
|
|
|
|
sched_bind(td, cpu);
|
|
|
|
thread_unlock(td);
|
|
|
|
|
|
|
|
for (domain = 0; domain < vm_ndomains; domain++) {
|
|
|
|
vmd = VM_DOMAIN(domain);
|
|
|
|
for (queue = 0; queue < PQ_COUNT; queue++) {
|
|
|
|
pq = &vmd->vmd_pagequeues[queue];
|
|
|
|
vm_pagequeue_lock(pq);
|
|
|
|
critical_enter();
|
|
|
|
vm_pqbatch_process(pq,
|
|
|
|
DPCPU_PTR(pqbatch[domain][queue]), queue);
|
|
|
|
critical_exit();
|
|
|
|
vm_pagequeue_unlock(pq);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
thread_lock(td);
|
|
|
|
sched_unbind(td);
|
|
|
|
thread_unlock(td);
|
|
|
|
}
|
|
|
|
|
2019-09-16 15:04:45 +00:00
|
|
|
/*
|
|
|
|
* vm_page_dequeue_deferred: [ internal use only ]
|
|
|
|
*
|
|
|
|
* Request removal of the given page from its current page
|
|
|
|
* queue. Physical removal from the queue may be deferred
|
|
|
|
* indefinitely.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_dequeue_deferred(vm_page_t m)
|
|
|
|
{
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_astate_t new, old;
|
2019-09-16 15:04:45 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
old = vm_page_astate_load(m);
|
|
|
|
do {
|
|
|
|
if (old.queue == PQ_NONE) {
|
|
|
|
KASSERT((old.flags & PGA_QUEUE_STATE_MASK) == 0,
|
|
|
|
("%s: page %p has unexpected queue state",
|
|
|
|
__func__, m));
|
2019-09-16 15:12:49 +00:00
|
|
|
break;
|
|
|
|
}
|
2019-12-28 19:03:32 +00:00
|
|
|
new = old;
|
|
|
|
new.flags |= PGA_DEQUEUE;
|
|
|
|
} while (!vm_page_pqstate_commit_request(m, &old, new));
|
2019-02-03 18:43:20 +00:00
|
|
|
}
|
|
|
|
|
2008-03-18 06:52:15 +00:00
|
|
|
/*
|
2018-04-24 21:15:54 +00:00
|
|
|
* vm_page_dequeue:
|
2008-03-18 06:52:15 +00:00
|
|
|
*
|
2019-12-28 19:03:32 +00:00
|
|
|
* Remove the page from whichever page queue it's in, if any, before
|
|
|
|
* returning.
|
2008-03-18 06:52:15 +00:00
|
|
|
*/
|
2018-04-24 21:15:54 +00:00
|
|
|
void
|
|
|
|
vm_page_dequeue(vm_page_t m)
|
2008-03-18 06:52:15 +00:00
|
|
|
{
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_astate_t new, old;
|
2019-09-16 15:04:45 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
old = vm_page_astate_load(m);
|
|
|
|
do {
|
|
|
|
if (old.queue == PQ_NONE) {
|
|
|
|
KASSERT((old.flags & PGA_QUEUE_STATE_MASK) == 0,
|
|
|
|
("%s: page %p has unexpected queue state",
|
|
|
|
__func__, m));
|
2018-04-24 21:15:54 +00:00
|
|
|
break;
|
2019-12-28 19:03:32 +00:00
|
|
|
}
|
|
|
|
new = old;
|
|
|
|
new.flags &= ~PGA_QUEUE_OP_MASK;
|
|
|
|
new.queue = PQ_NONE;
|
|
|
|
} while (!vm_page_pqstate_commit_dequeue(m, &old, new));
|
|
|
|
|
2012-11-13 02:50:39 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2018-04-24 21:15:54 +00:00
|
|
|
* Schedule the given page for insertion into the specified page queue.
|
|
|
|
* Physical insertion of the page may be deferred indefinitely.
|
2012-11-13 02:50:39 +00:00
|
|
|
*/
|
2018-04-24 21:15:54 +00:00
|
|
|
static void
|
|
|
|
vm_page_enqueue(vm_page_t m, uint8_t queue)
|
2012-11-13 02:50:39 +00:00
|
|
|
{
|
|
|
|
|
2019-12-10 18:14:50 +00:00
|
|
|
KASSERT(m->a.queue == PQ_NONE &&
|
|
|
|
(m->a.flags & PGA_QUEUE_STATE_MASK) == 0,
|
2018-04-24 21:15:54 +00:00
|
|
|
("%s: page %p is already enqueued", __func__, m));
|
2019-11-06 16:59:16 +00:00
|
|
|
KASSERT(m->ref_count > 0,
|
|
|
|
("%s: page %p does not carry any references", __func__, m));
|
2018-04-24 21:15:54 +00:00
|
|
|
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.queue = queue;
|
|
|
|
if ((m->a.flags & PGA_REQUEUE) == 0)
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_page_aflag_set(m, PGA_REQUEUE);
|
2019-08-23 19:49:29 +00:00
|
|
|
vm_page_pqbatch_submit(m, queue);
|
2012-11-13 02:50:39 +00:00
|
|
|
}
|
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
2017-09-13 19:11:52 +00:00
|
|
|
* vm_page_free_prep:
|
1999-01-21 08:29:12 +00:00
|
|
|
*
|
2017-09-13 19:11:52 +00:00
|
|
|
* Prepares the given page to be put on the free list,
|
|
|
|
* disassociating it from any VM object. The caller may return
|
|
|
|
* the page to the free list only if this function returns true.
|
1999-01-21 08:29:12 +00:00
|
|
|
*
|
2020-06-27 02:31:39 +00:00
|
|
|
* The object, if it exists, must be locked, and then the page must
|
|
|
|
* be xbusy. Otherwise the page must be not busied. A managed
|
|
|
|
* page must be unmapped.
|
1999-01-21 08:29:12 +00:00
|
|
|
*/
|
2019-12-22 06:56:44 +00:00
|
|
|
static bool
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_page_free_prep(vm_page_t m)
|
1999-01-21 08:29:12 +00:00
|
|
|
{
|
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
/*
|
|
|
|
* Synchronize with threads that have dropped a reference to this
|
|
|
|
* page.
|
|
|
|
*/
|
|
|
|
atomic_thread_fence_acq();
|
|
|
|
|
2017-10-21 17:28:12 +00:00
|
|
|
#if defined(DIAGNOSTIC) && defined(PHYS_TO_DMAP)
|
Remove SFBUF_OPTIONAL_DIRECT_MAP and such hacks, replacing them across the
kernel by PHYS_TO_DMAP() as previously present on amd64, arm64, riscv, and
powerpc64. This introduces a new MI macro (PMAP_HAS_DMAP) that can be
evaluated at runtime to determine if the architecture has a direct map;
if it does not (or does) unconditionally and PMAP_HAS_DMAP is either 0 or
1, the compiler can remove the conditional logic.
As part of this, implement PHYS_TO_DMAP() on sparc64 and mips64, which had
similar things but spelled differently. 32-bit MIPS has a partial direct-map
that maps poorly to this concept and is unchanged.
Reviewed by: kib
Suggestions from: marius, alc, kib
Runtime tested on: amd64, powerpc64, powerpc, mips64
2018-01-19 17:46:31 +00:00
|
|
|
if (PMAP_HAS_DMAP && (m->flags & PG_ZERO) != 0) {
|
2017-10-21 17:28:12 +00:00
|
|
|
uint64_t *p;
|
|
|
|
int i;
|
|
|
|
p = (uint64_t *)PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m));
|
|
|
|
for (i = 0; i < PAGE_SIZE / sizeof(uint64_t); i++, p++)
|
|
|
|
KASSERT(*p == 0, ("vm_page_free_prep %p PG_ZERO %d %jx",
|
|
|
|
m, i, (uintmax_t)*p));
|
|
|
|
}
|
|
|
|
#endif
|
2019-10-07 23:31:17 +00:00
|
|
|
if ((m->oflags & VPO_UNMANAGED) == 0) {
|
2019-09-16 15:04:45 +00:00
|
|
|
KASSERT(!pmap_page_is_mapped(m),
|
2018-04-24 21:15:54 +00:00
|
|
|
("vm_page_free_prep: freeing mapped page %p", m));
|
2019-12-10 18:14:50 +00:00
|
|
|
KASSERT((m->a.flags & (PGA_EXECUTABLE | PGA_WRITEABLE)) == 0,
|
2019-10-07 23:31:17 +00:00
|
|
|
("vm_page_free_prep: mapping flags set in page %p", m));
|
|
|
|
} else {
|
2019-12-10 18:14:50 +00:00
|
|
|
KASSERT(m->a.queue == PQ_NONE,
|
2018-04-24 21:15:54 +00:00
|
|
|
("vm_page_free_prep: unmanaged page %p is queued", m));
|
2019-10-07 23:31:17 +00:00
|
|
|
}
|
- Remove 'struct vmmeter' from 'struct pcpu', leaving only global vmmeter
in place. To do per-cpu stats, convert all fields that previously were
maintained in the vmmeters that sit in pcpus to counter(9).
- Since some vmmeter stats may be touched at very early stages of boot,
before we have set up UMA and we can do counter_u64_alloc(), provide an
early counter mechanism:
o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter.
o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter,
so that at early stages of boot, before counters are allocated we already
point to a counter that can be safely written to.
o For sparc64 that required a whole dummy pcpu[MAXCPU] array.
Further related changes:
- Don't include vmmeter.h into pcpu.h.
- vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit,
to match kernel representation.
- struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion.
This is based on benno@'s 4-year old patch:
https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html
Reviewed by: kib, gallatin, marius, lidl
Differential Revision: https://reviews.freebsd.org/D10156
2017-04-17 17:34:47 +00:00
|
|
|
VM_CNT_INC(v_tfree);
|
1998-01-17 09:17:02 +00:00
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
if (m->object != NULL) {
|
2019-12-21 19:04:05 +00:00
|
|
|
KASSERT(((m->oflags & VPO_UNMANAGED) != 0) ==
|
|
|
|
((m->object->flags & OBJ_UNMANAGED) != 0),
|
|
|
|
("vm_page_free_prep: managed flag mismatch for page %p",
|
|
|
|
m));
|
2020-02-04 20:33:01 +00:00
|
|
|
vm_page_assert_xbusied(m);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The object reference can be released without an atomic
|
|
|
|
* operation.
|
|
|
|
*/
|
|
|
|
KASSERT((m->flags & PG_FICTITIOUS) != 0 ||
|
|
|
|
m->ref_count == VPRC_OBJREF,
|
|
|
|
("vm_page_free_prep: page %p has unexpected ref_count %u",
|
|
|
|
m, m->ref_count));
|
2020-02-04 20:33:01 +00:00
|
|
|
vm_page_object_remove(m);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
m->ref_count -= VPRC_OBJREF;
|
2020-02-04 20:33:01 +00:00
|
|
|
} else
|
|
|
|
vm_page_assert_unbusied(m);
|
2017-10-24 17:14:53 +00:00
|
|
|
|
2020-02-04 20:33:01 +00:00
|
|
|
vm_page_busy_free(m);
|
2019-10-15 03:38:02 +00:00
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
2017-10-24 17:14:53 +00:00
|
|
|
* If fictitious remove object association and
|
|
|
|
* return.
|
1999-01-21 08:29:12 +00:00
|
|
|
*/
|
2017-10-24 17:14:53 +00:00
|
|
|
if ((m->flags & PG_FICTITIOUS) != 0) {
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT(m->ref_count == 1,
|
|
|
|
("fictitious page %p is referenced", m));
|
2019-12-10 18:14:50 +00:00
|
|
|
KASSERT(m->a.queue == PQ_NONE,
|
2017-10-24 17:14:53 +00:00
|
|
|
("fictitious page %p is queued", m));
|
|
|
|
return (false);
|
|
|
|
}
|
|
|
|
|
2018-04-24 21:15:54 +00:00
|
|
|
/*
|
|
|
|
* Pages need not be dequeued before they are returned to the physical
|
|
|
|
* memory allocator, but they must at least be marked for a deferred
|
|
|
|
* dequeue.
|
|
|
|
*/
|
|
|
|
if ((m->oflags & VPO_UNMANAGED) == 0)
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_dequeue_deferred(m);
|
2018-04-24 21:15:54 +00:00
|
|
|
|
1998-03-01 04:18:54 +00:00
|
|
|
m->valid = 0;
|
1999-08-17 05:08:39 +00:00
|
|
|
vm_page_undirty(m);
|
1998-03-01 04:18:54 +00:00
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
if (m->ref_count != 0)
|
|
|
|
panic("vm_page_free_prep: page %p has references", m);
|
2009-07-12 23:31:20 +00:00
|
|
|
|
2017-09-13 19:11:52 +00:00
|
|
|
/*
|
|
|
|
* Restore the default memory attribute to the page.
|
|
|
|
*/
|
|
|
|
if (pmap_page_get_memattr(m) != VM_MEMATTR_DEFAULT)
|
|
|
|
pmap_page_set_memattr(m, VM_MEMATTR_DEFAULT);
|
|
|
|
|
2007-12-29 19:53:04 +00:00
|
|
|
#if VM_NRESERVLEVEL > 0
|
2019-07-08 19:02:40 +00:00
|
|
|
/*
|
|
|
|
* Determine whether the page belongs to a reservation. If the page was
|
|
|
|
* allocated from a per-CPU cache, it cannot belong to a reservation, so
|
|
|
|
* as an optimization, we avoid the check in that case.
|
|
|
|
*/
|
|
|
|
if ((m->flags & PG_PCPU_CACHE) == 0 && vm_reserv_free_page(m))
|
2018-03-22 19:21:11 +00:00
|
|
|
return (false);
|
2007-12-29 19:53:04 +00:00
|
|
|
#endif
|
2018-03-22 19:21:11 +00:00
|
|
|
|
|
|
|
return (true);
|
2017-09-13 19:11:52 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_free_toq:
|
|
|
|
*
|
|
|
|
* Returns the given page to the free list, disassociating it
|
|
|
|
* from any VM object.
|
|
|
|
*
|
2020-08-04 14:59:43 +00:00
|
|
|
* The object must be locked. The page must be exclusively busied if it
|
|
|
|
* belongs to an object.
|
2017-09-13 19:11:52 +00:00
|
|
|
*/
|
2019-12-22 06:56:44 +00:00
|
|
|
static void
|
2017-09-13 19:11:52 +00:00
|
|
|
vm_page_free_toq(vm_page_t m)
|
|
|
|
{
|
2018-02-06 22:10:07 +00:00
|
|
|
struct vm_domain *vmd;
|
2019-07-08 18:56:30 +00:00
|
|
|
uma_zone_t zone;
|
2017-09-13 19:11:52 +00:00
|
|
|
|
2018-04-24 21:15:54 +00:00
|
|
|
if (!vm_page_free_prep(m))
|
2017-09-13 19:11:52 +00:00
|
|
|
return;
|
2018-04-01 04:50:05 +00:00
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
vmd = vm_pagequeue_domain(m);
|
2019-07-08 18:56:30 +00:00
|
|
|
zone = vmd->vmd_pgcache[m->pool].zone;
|
|
|
|
if ((m->flags & PG_PCPU_CACHE) != 0 && zone != NULL) {
|
|
|
|
uma_zfree(zone, m);
|
2018-04-01 04:50:05 +00:00
|
|
|
return;
|
|
|
|
}
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_domain_free_lock(vmd);
|
2018-03-22 19:21:11 +00:00
|
|
|
vm_phys_free_pages(m, 0);
|
2018-02-06 22:10:07 +00:00
|
|
|
vm_domain_free_unlock(vmd);
|
2018-03-15 19:23:07 +00:00
|
|
|
vm_domain_freecnt_inc(vmd, 1);
|
1996-06-05 03:31:49 +00:00
|
|
|
}
|
1994-05-25 09:21:21 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
2018-03-04 20:53:20 +00:00
|
|
|
* vm_page_free_pages_toq:
|
|
|
|
*
|
|
|
|
* Returns a list of pages to the free list, disassociating it
|
|
|
|
* from any VM object. In other words, this is equivalent to
|
|
|
|
* calling vm_page_free_toq() for each page of a list of VM objects.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_free_pages_toq(struct spglist *free, bool update_wire_count)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
int count;
|
|
|
|
|
|
|
|
if (SLIST_EMPTY(free))
|
|
|
|
return;
|
|
|
|
|
|
|
|
count = 0;
|
|
|
|
while ((m = SLIST_FIRST(free)) != NULL) {
|
|
|
|
count++;
|
|
|
|
SLIST_REMOVE_HEAD(free, plinks.s.ss);
|
2018-04-01 04:50:05 +00:00
|
|
|
vm_page_free_toq(m);
|
2018-03-04 20:53:20 +00:00
|
|
|
}
|
|
|
|
|
2018-03-04 21:15:31 +00:00
|
|
|
if (update_wire_count)
|
2018-03-04 20:53:20 +00:00
|
|
|
vm_wire_sub(count);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2020-08-25 13:45:06 +00:00
|
|
|
* Mark this page as wired down. For managed pages, this prevents reclamation
|
|
|
|
* by the page daemon, or when the containing object, if any, is destroyed.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_wire(vm_page_t m)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
u_int old;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2020-08-25 13:45:06 +00:00
|
|
|
#ifdef INVARIANTS
|
|
|
|
if (m->object != NULL && !vm_page_busied(m) &&
|
|
|
|
!vm_object_busied(m->object))
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(m->object);
|
2020-08-25 13:45:06 +00:00
|
|
|
#endif
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT((m->flags & PG_FICTITIOUS) == 0 ||
|
|
|
|
VPRC_WIRE_COUNT(m->ref_count) >= 1,
|
|
|
|
("vm_page_wire: fictitious page %p has zero wirings", m));
|
|
|
|
|
|
|
|
old = atomic_fetchadd_int(&m->ref_count, 1);
|
|
|
|
KASSERT(VPRC_WIRE_COUNT(old) != VPRC_WIRE_COUNT_MAX,
|
|
|
|
("vm_page_wire: counter overflow for page %p", m));
|
2019-12-28 19:03:46 +00:00
|
|
|
if (VPRC_WIRE_COUNT(old) == 0) {
|
|
|
|
if ((m->oflags & VPO_UNMANAGED) == 0)
|
|
|
|
vm_page_aflag_set(m, PGA_DEQUEUE);
|
2018-02-12 22:53:00 +00:00
|
|
|
vm_wire_add(1);
|
2019-12-28 19:03:46 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
* Attempt to wire a mapped page following a pmap lookup of that page.
|
|
|
|
* This may fail if a thread is concurrently tearing down mappings of the page.
|
2019-10-30 17:33:17 +00:00
|
|
|
* The transient failure is acceptable because it translates to the
|
|
|
|
* failure of the caller pmap_extract_and_hold(), which should be then
|
|
|
|
* followed by the vm_fault() fallback, see e.g. vm_fault_quick_hold_pages().
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
*/
|
|
|
|
bool
|
|
|
|
vm_page_wire_mapped(vm_page_t m)
|
|
|
|
{
|
|
|
|
u_int old;
|
|
|
|
|
|
|
|
old = m->ref_count;
|
|
|
|
do {
|
|
|
|
KASSERT(old > 0,
|
|
|
|
("vm_page_wire_mapped: wiring unreferenced page %p", m));
|
|
|
|
if ((old & VPRC_BLOCKED) != 0)
|
|
|
|
return (false);
|
|
|
|
} while (!atomic_fcmpset_int(&m->ref_count, &old, old + 1));
|
|
|
|
|
2019-12-28 19:03:46 +00:00
|
|
|
if (VPRC_WIRE_COUNT(old) == 0) {
|
|
|
|
if ((m->oflags & VPO_UNMANAGED) == 0)
|
|
|
|
vm_page_aflag_set(m, PGA_DEQUEUE);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
vm_wire_add(1);
|
2019-12-28 19:03:46 +00:00
|
|
|
}
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
return (true);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2019-12-28 19:03:32 +00:00
|
|
|
* Release a wiring reference to a managed page. If the page still belongs to
|
|
|
|
* an object, update its position in the page queues to reflect the reference.
|
|
|
|
* If the wiring was the last reference to the page, free the page.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2019-12-28 19:03:32 +00:00
|
|
|
static void
|
|
|
|
vm_page_unwire_managed(vm_page_t m, uint8_t nqueue, bool noreuse)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2019-09-16 15:04:45 +00:00
|
|
|
u_int old;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
|
|
|
|
("%s: page %p is unmanaged", __func__, m));
|
2019-09-16 15:04:45 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Update LRU state before releasing the wiring reference.
|
|
|
|
* Use a release store when updating the reference count to
|
|
|
|
* synchronize with vm_page_free_prep().
|
|
|
|
*/
|
|
|
|
old = m->ref_count;
|
|
|
|
do {
|
|
|
|
KASSERT(VPRC_WIRE_COUNT(old) > 0,
|
|
|
|
("vm_page_unwire: wire count underflow for page %p", m));
|
2019-12-28 19:03:46 +00:00
|
|
|
|
|
|
|
if (old > VPRC_OBJREF + 1) {
|
|
|
|
/*
|
|
|
|
* The page has at least one other wiring reference. An
|
|
|
|
* earlier iteration of this loop may have called
|
|
|
|
* vm_page_release_toq() and cleared PGA_DEQUEUE, so
|
|
|
|
* re-set it if necessary.
|
|
|
|
*/
|
|
|
|
if ((vm_page_astate_load(m).flags & PGA_DEQUEUE) == 0)
|
|
|
|
vm_page_aflag_set(m, PGA_DEQUEUE);
|
|
|
|
} else if (old == VPRC_OBJREF + 1) {
|
|
|
|
/*
|
|
|
|
* This is the last wiring. Clear PGA_DEQUEUE and
|
|
|
|
* update the page's queue state to reflect the
|
|
|
|
* reference. If the page does not belong to an object
|
|
|
|
* (i.e., the VPRC_OBJREF bit is clear), we only need to
|
|
|
|
* clear leftover queue state.
|
|
|
|
*/
|
2021-02-10 16:10:27 +00:00
|
|
|
vm_page_release_toq(m, nqueue, noreuse);
|
2019-12-28 19:03:46 +00:00
|
|
|
} else if (old == 1) {
|
|
|
|
vm_page_aflag_clear(m, PGA_DEQUEUE);
|
2019-09-16 15:04:45 +00:00
|
|
|
}
|
|
|
|
} while (!atomic_fcmpset_rel_int(&m->ref_count, &old, old - 1));
|
|
|
|
|
|
|
|
if (VPRC_WIRE_COUNT(old) == 1) {
|
|
|
|
vm_wire_sub(1);
|
|
|
|
if (old == 1)
|
|
|
|
vm_page_free(m);
|
Dequeue wired pages lazily.
Previously, wiring a page would cause it to be removed from its page
queue. In the common case, unwiring causes it to be enqueued at the tail
of that page queue. This change modifies vm_page_wire() to not dequeue
the page, thus avoiding the highly contended page queue locks. Instead,
vm_page_unwire() takes care of requeuing the page as a single operation,
and the page daemon dequeues wired pages as they are encountered during
a queue scan to avoid needlessly revisiting them later. For pages in
PQ_ACTIVE we do even better, since a requeue is unnecessary.
The change improves scalability for some common workloads. For instance,
threads wiring pages into the buffer cache no longer need to modify
global page queues, and unwiring is usually done by the bufspace thread,
so concurrency is not as much of an issue. As another example, many
sysctl handlers wire the output buffer to avoid faults on copyout, and
since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid
modifying the page queue in this case.
The change also adds a block comment describing some properties of
struct vm_page's reference counters, and the busy lock.
Reviewed by: jeff
Discussed with: alc, kib
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
/*
|
|
|
|
* Release one wiring of the specified page, potentially allowing it to be
|
|
|
|
* paged out.
|
|
|
|
*
|
|
|
|
* Only managed pages belonging to an object can be paged out. If the number
|
|
|
|
* of wirings transitions to zero and the page is eligible for page out, then
|
|
|
|
* the page is added to the specified paging queue. If the released wiring
|
|
|
|
* represented the last reference to the page, the page is freed.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_unwire(vm_page_t m, uint8_t nqueue)
|
|
|
|
{
|
|
|
|
|
|
|
|
KASSERT(nqueue < PQ_COUNT,
|
|
|
|
("vm_page_unwire: invalid queue %u request for page %p",
|
|
|
|
nqueue, m));
|
|
|
|
|
|
|
|
if ((m->oflags & VPO_UNMANAGED) != 0) {
|
|
|
|
if (vm_page_unwire_noq(m) && m->ref_count == 0)
|
|
|
|
vm_page_free(m);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
vm_page_unwire_managed(m, nqueue, false);
|
|
|
|
}
|
|
|
|
|
Dequeue wired pages lazily.
Previously, wiring a page would cause it to be removed from its page
queue. In the common case, unwiring causes it to be enqueued at the tail
of that page queue. This change modifies vm_page_wire() to not dequeue
the page, thus avoiding the highly contended page queue locks. Instead,
vm_page_unwire() takes care of requeuing the page as a single operation,
and the page daemon dequeues wired pages as they are encountered during
a queue scan to avoid needlessly revisiting them later. For pages in
PQ_ACTIVE we do even better, since a requeue is unnecessary.
The change improves scalability for some common workloads. For instance,
threads wiring pages into the buffer cache no longer need to modify
global page queues, and unwiring is usually done by the bufspace thread,
so concurrency is not as much of an issue. As another example, many
sysctl handlers wire the output buffer to avoid faults on copyout, and
since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid
modifying the page queue in this case.
The change also adds a block comment describing some properties of
struct vm_page's reference counters, and the busy lock.
Reviewed by: jeff
Discussed with: alc, kib
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
|
|
|
/*
|
|
|
|
* Unwire a page without (re-)inserting it into a page queue. It is up
|
|
|
|
* to the caller to enqueue, requeue, or free the page as appropriate.
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
* In most cases involving managed pages, vm_page_unwire() should be used
|
|
|
|
* instead.
|
Dequeue wired pages lazily.
Previously, wiring a page would cause it to be removed from its page
queue. In the common case, unwiring causes it to be enqueued at the tail
of that page queue. This change modifies vm_page_wire() to not dequeue
the page, thus avoiding the highly contended page queue locks. Instead,
vm_page_unwire() takes care of requeuing the page as a single operation,
and the page daemon dequeues wired pages as they are encountered during
a queue scan to avoid needlessly revisiting them later. For pages in
PQ_ACTIVE we do even better, since a requeue is unnecessary.
The change improves scalability for some common workloads. For instance,
threads wiring pages into the buffer cache no longer need to modify
global page queues, and unwiring is usually done by the bufspace thread,
so concurrency is not as much of an issue. As another example, many
sysctl handlers wire the output buffer to avoid faults on copyout, and
since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid
modifying the page queue in this case.
The change also adds a block comment describing some properties of
struct vm_page's reference counters, and the busy lock.
Reviewed by: jeff
Discussed with: alc, kib
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
|
|
|
*/
|
|
|
|
bool
|
|
|
|
vm_page_unwire_noq(vm_page_t m)
|
|
|
|
{
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
u_int old;
|
Dequeue wired pages lazily.
Previously, wiring a page would cause it to be removed from its page
queue. In the common case, unwiring causes it to be enqueued at the tail
of that page queue. This change modifies vm_page_wire() to not dequeue
the page, thus avoiding the highly contended page queue locks. Instead,
vm_page_unwire() takes care of requeuing the page as a single operation,
and the page daemon dequeues wired pages as they are encountered during
a queue scan to avoid needlessly revisiting them later. For pages in
PQ_ACTIVE we do even better, since a requeue is unnecessary.
The change improves scalability for some common workloads. For instance,
threads wiring pages into the buffer cache no longer need to modify
global page queues, and unwiring is usually done by the bufspace thread,
so concurrency is not as much of an issue. As another example, many
sysctl handlers wire the output buffer to avoid faults on copyout, and
since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid
modifying the page queue in this case.
The change also adds a block comment describing some properties of
struct vm_page's reference counters, and the busy lock.
Reviewed by: jeff
Discussed with: alc, kib
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
old = vm_page_drop(m, 1);
|
|
|
|
KASSERT(VPRC_WIRE_COUNT(old) != 0,
|
2021-08-22 17:21:01 +00:00
|
|
|
("%s: counter underflow for page %p", __func__, m));
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
KASSERT((m->flags & PG_FICTITIOUS) == 0 || VPRC_WIRE_COUNT(old) > 1,
|
2021-08-22 17:21:01 +00:00
|
|
|
("%s: missing ref on fictitious page %p", __func__, m));
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
|
|
|
|
if (VPRC_WIRE_COUNT(old) > 1)
|
Dequeue wired pages lazily.
Previously, wiring a page would cause it to be removed from its page
queue. In the common case, unwiring causes it to be enqueued at the tail
of that page queue. This change modifies vm_page_wire() to not dequeue
the page, thus avoiding the highly contended page queue locks. Instead,
vm_page_unwire() takes care of requeuing the page as a single operation,
and the page daemon dequeues wired pages as they are encountered during
a queue scan to avoid needlessly revisiting them later. For pages in
PQ_ACTIVE we do even better, since a requeue is unnecessary.
The change improves scalability for some common workloads. For instance,
threads wiring pages into the buffer cache no longer need to modify
global page queues, and unwiring is usually done by the bufspace thread,
so concurrency is not as much of an issue. As another example, many
sysctl handlers wire the output buffer to avoid faults on copyout, and
since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid
modifying the page queue in this case.
The change also adds a block comment describing some properties of
struct vm_page's reference counters, and the busy lock.
Reviewed by: jeff
Discussed with: alc, kib
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
|
|
|
return (false);
|
2019-12-28 19:03:46 +00:00
|
|
|
if ((m->oflags & VPO_UNMANAGED) == 0)
|
|
|
|
vm_page_aflag_clear(m, PGA_DEQUEUE);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
vm_wire_sub(1);
|
|
|
|
return (true);
|
1995-03-25 08:47:35 +00:00
|
|
|
}
|
|
|
|
|
2019-06-19 21:36:00 +00:00
|
|
|
/*
|
2019-12-28 19:03:32 +00:00
|
|
|
* Ensure that the page ends up in the specified page queue. If the page is
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
* active or being moved to the active queue, ensure that its act_count is
|
2019-12-28 19:03:32 +00:00
|
|
|
* at least ACT_INIT but do not otherwise mess with it.
|
2019-06-19 21:36:00 +00:00
|
|
|
*/
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
static __always_inline void
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_mvqueue(vm_page_t m, const uint8_t nqueue, const uint16_t nflag)
|
2019-06-19 21:36:00 +00:00
|
|
|
{
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_astate_t old, new;
|
2019-06-19 21:36:00 +00:00
|
|
|
|
2019-11-06 16:59:16 +00:00
|
|
|
KASSERT(m->ref_count > 0,
|
|
|
|
("%s: page %p does not carry any references", __func__, m));
|
2019-12-28 19:03:32 +00:00
|
|
|
KASSERT(nflag == PGA_REQUEUE || nflag == PGA_REQUEUE_HEAD,
|
|
|
|
("%s: invalid flags %x", __func__, nflag));
|
2019-09-16 15:04:45 +00:00
|
|
|
|
2019-12-29 20:01:03 +00:00
|
|
|
if ((m->oflags & VPO_UNMANAGED) != 0 || vm_page_wired(m))
|
2019-12-28 19:03:46 +00:00
|
|
|
return;
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
old = vm_page_astate_load(m);
|
|
|
|
do {
|
2019-12-28 19:03:46 +00:00
|
|
|
if ((old.flags & PGA_DEQUEUE) != 0)
|
|
|
|
break;
|
2019-12-28 19:03:32 +00:00
|
|
|
new = old;
|
2019-12-29 15:39:43 +00:00
|
|
|
new.flags &= ~PGA_QUEUE_OP_MASK;
|
2019-12-28 19:03:32 +00:00
|
|
|
if (nqueue == PQ_ACTIVE)
|
|
|
|
new.act_count = max(old.act_count, ACT_INIT);
|
|
|
|
if (old.queue == nqueue) {
|
vm_page: Fix a logic error in the handling of PQ_ACTIVE operations
As an optimization, vm_page_activate() avoids requeuing a page that's
already in the active queue. A page's location in the active queue is
mostly unimportant.
When a page is unwired and placed back in the page queues,
vm_page_unwire() avoids moving pages out of PQ_ACTIVE to honour the
request, the idea being that they're likely mapped and so will simply
get bounced back in to PQ_ACTIVE during a queue scan.
In both cases, if the page was logically in PQ_ACTIVE but had not yet
been physically enqueued (i.e., the page is in a per-CPU batch), we
would end up clearing PGA_REQUEUE from the page. Then, batch processing
would ignore the page, so it would end up unwired and not in any queues.
This can arise, for example, when a page is allocated and then
vm_page_activate() is called multiple times in quick succession. The
result is that the page is hidden from the page daemon, so while it will
be freed when its VM object is destroyed, it cannot be reclaimed under
memory pressure.
Fix the bug: when checking if a page is in PQ_ACTIVE, only perform the
optimization if the page is physically enqueued.
PR: 256507
Fixes: f3f38e2580f1 ("Start implementing queue state updates using fcmpset loops.")
Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: E-CARD Ltd.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36839
2022-10-05 19:12:46 +00:00
|
|
|
/*
|
|
|
|
* There is no need to requeue pages already in the
|
|
|
|
* active queue.
|
|
|
|
*/
|
|
|
|
if (nqueue != PQ_ACTIVE ||
|
|
|
|
(old.flags & PGA_ENQUEUED) == 0)
|
2019-12-28 19:03:32 +00:00
|
|
|
new.flags |= nflag;
|
|
|
|
} else {
|
|
|
|
new.flags |= nflag;
|
|
|
|
new.queue = nqueue;
|
|
|
|
}
|
|
|
|
} while (!vm_page_pqstate_commit(m, &old, new));
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Put the specified page on the active list (if appropriate).
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_activate(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_mvqueue(m, PQ_ACTIVE, PGA_REQUEUE);
|
2019-06-19 21:36:00 +00:00
|
|
|
}
|
|
|
|
|
1994-05-25 09:21:21 +00:00
|
|
|
/*
|
2018-04-24 21:15:54 +00:00
|
|
|
* Move the specified page to the tail of the inactive queue, or requeue
|
|
|
|
* the page if it is already in the inactive queue.
|
1994-05-25 09:21:21 +00:00
|
|
|
*/
|
2018-04-24 21:15:54 +00:00
|
|
|
void
|
|
|
|
vm_page_deactivate(vm_page_t m)
|
1994-05-25 09:21:21 +00:00
|
|
|
{
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_mvqueue(m, PQ_INACTIVE, PGA_REQUEUE);
|
2015-09-30 23:06:29 +00:00
|
|
|
}
|
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
void
|
|
|
|
vm_page_deactivate_noreuse(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_mvqueue(m, PQ_INACTIVE, PGA_REQUEUE_HEAD);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
}
|
|
|
|
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
/*
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
* Put a page in the laundry, or requeue it if it is already there.
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
*/
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
void
|
|
|
|
vm_page_launder(vm_page_t m)
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
{
|
2001-05-19 01:28:09 +00:00
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_mvqueue(m, PQ_LAUNDRY, PGA_REQUEUE);
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
}
|
|
|
|
|
2017-01-03 00:05:44 +00:00
|
|
|
/*
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
* Put a page in the PQ_UNSWAPPABLE holding queue.
|
2017-01-03 00:05:44 +00:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_unswappable(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
2022-11-09 20:13:01 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(m->object);
|
|
|
|
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
|
2017-01-03 00:05:44 +00:00
|
|
|
("page %p already unswappable", m));
|
2018-04-24 21:15:54 +00:00
|
|
|
|
2018-09-06 16:17:45 +00:00
|
|
|
vm_page_dequeue(m);
|
2018-04-24 21:15:54 +00:00
|
|
|
vm_page_enqueue(m, PQ_UNSWAPPABLE);
|
2017-01-03 00:05:44 +00:00
|
|
|
}
|
|
|
|
|
2019-12-28 19:03:32 +00:00
|
|
|
/*
|
|
|
|
* Release a page back to the page queues in preparation for unwiring.
|
|
|
|
*/
|
2019-09-16 15:04:45 +00:00
|
|
|
static void
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_release_toq(vm_page_t m, uint8_t nqueue, const bool noreuse)
|
2019-07-29 22:01:28 +00:00
|
|
|
{
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_astate_t old, new;
|
|
|
|
uint16_t nflag;
|
2019-07-29 22:01:28 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Use a check of the valid bits to determine whether we should
|
|
|
|
* accelerate reclamation of the page. The object lock might not be
|
|
|
|
* held here, in which case the check is racy. At worst we will either
|
|
|
|
* accelerate reclamation of a valid page and violate LRU, or
|
|
|
|
* unnecessarily defer reclamation of an invalid page.
|
|
|
|
*
|
|
|
|
* If we were asked to not cache the page, place it near the head of the
|
|
|
|
* inactive queue so that is reclaimed sooner.
|
|
|
|
*/
|
2022-10-18 09:16:36 +00:00
|
|
|
if (noreuse || vm_page_none_valid(m)) {
|
2019-12-28 19:03:32 +00:00
|
|
|
nqueue = PQ_INACTIVE;
|
|
|
|
nflag = PGA_REQUEUE_HEAD;
|
|
|
|
} else {
|
|
|
|
nflag = PGA_REQUEUE;
|
|
|
|
}
|
|
|
|
|
|
|
|
old = vm_page_astate_load(m);
|
|
|
|
do {
|
|
|
|
new = old;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the page is already in the active queue and we are not
|
|
|
|
* trying to accelerate reclamation, simply mark it as
|
|
|
|
* referenced and avoid any queue operations.
|
|
|
|
*/
|
|
|
|
new.flags &= ~PGA_QUEUE_OP_MASK;
|
vm_page: Fix a logic error in the handling of PQ_ACTIVE operations
As an optimization, vm_page_activate() avoids requeuing a page that's
already in the active queue. A page's location in the active queue is
mostly unimportant.
When a page is unwired and placed back in the page queues,
vm_page_unwire() avoids moving pages out of PQ_ACTIVE to honour the
request, the idea being that they're likely mapped and so will simply
get bounced back in to PQ_ACTIVE during a queue scan.
In both cases, if the page was logically in PQ_ACTIVE but had not yet
been physically enqueued (i.e., the page is in a per-CPU batch), we
would end up clearing PGA_REQUEUE from the page. Then, batch processing
would ignore the page, so it would end up unwired and not in any queues.
This can arise, for example, when a page is allocated and then
vm_page_activate() is called multiple times in quick succession. The
result is that the page is hidden from the page daemon, so while it will
be freed when its VM object is destroyed, it cannot be reclaimed under
memory pressure.
Fix the bug: when checking if a page is in PQ_ACTIVE, only perform the
optimization if the page is physically enqueued.
PR: 256507
Fixes: f3f38e2580f1 ("Start implementing queue state updates using fcmpset loops.")
Reviewed by: alc, kib
MFC after: 1 week
Sponsored by: E-CARD Ltd.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36839
2022-10-05 19:12:46 +00:00
|
|
|
if (nflag != PGA_REQUEUE_HEAD && old.queue == PQ_ACTIVE &&
|
|
|
|
(old.flags & PGA_ENQUEUED) != 0)
|
2019-12-28 19:03:32 +00:00
|
|
|
new.flags |= PGA_REFERENCED;
|
|
|
|
else {
|
|
|
|
new.flags |= nflag;
|
|
|
|
new.queue = nqueue;
|
|
|
|
}
|
|
|
|
} while (!vm_page_pqstate_commit(m, &old, new));
|
2019-07-29 22:01:28 +00:00
|
|
|
}
|
|
|
|
|
2001-05-24 07:22:27 +00:00
|
|
|
/*
|
2019-07-29 22:01:28 +00:00
|
|
|
* Unwire a page and either attempt to free it or re-add it to the page queues.
|
2001-05-24 07:22:27 +00:00
|
|
|
*/
|
2019-07-29 22:01:28 +00:00
|
|
|
void
|
|
|
|
vm_page_release(vm_page_t m, int flags)
|
|
|
|
{
|
|
|
|
vm_object_t object;
|
|
|
|
|
|
|
|
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
|
|
|
|
("vm_page_release: page %p is unmanaged", m));
|
|
|
|
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
if ((flags & VPR_TRYFREE) != 0) {
|
|
|
|
for (;;) {
|
2020-02-14 23:16:29 +00:00
|
|
|
object = atomic_load_ptr(&m->object);
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
if (object == NULL)
|
|
|
|
break;
|
|
|
|
/* Depends on type-stability. */
|
2019-12-22 06:56:44 +00:00
|
|
|
if (vm_page_busied(m) || !VM_OBJECT_TRYWLOCK(object))
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
break;
|
2019-12-22 06:56:44 +00:00
|
|
|
if (object == m->object) {
|
|
|
|
vm_page_release_locked(m, flags);
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
return;
|
2019-07-29 22:01:28 +00:00
|
|
|
}
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
}
|
|
|
|
}
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_unwire_managed(m, PQ_INACTIVE, flags != 0);
|
2019-07-29 22:01:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/* See vm_page_release(). */
|
|
|
|
void
|
|
|
|
vm_page_release_locked(vm_page_t m, int flags)
|
2001-05-24 07:22:27 +00:00
|
|
|
{
|
2002-07-20 20:12:57 +00:00
|
|
|
|
2017-09-24 23:35:01 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(m->object);
|
2019-07-29 22:01:28 +00:00
|
|
|
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
|
|
|
|
("vm_page_release_locked: page %p is unmanaged", m));
|
|
|
|
|
|
|
|
if (vm_page_unwire_noq(m)) {
|
|
|
|
if ((flags & VPR_TRYFREE) != 0 &&
|
|
|
|
(m->object->ref_count == 0 || !pmap_page_is_mapped(m)) &&
|
2019-12-22 06:56:44 +00:00
|
|
|
m->dirty == 0 && vm_page_tryxbusy(m)) {
|
2020-04-28 13:51:41 +00:00
|
|
|
/*
|
|
|
|
* An unlocked lookup may have wired the page before the
|
|
|
|
* busy lock was acquired, in which case the page must
|
|
|
|
* not be freed.
|
|
|
|
*/
|
|
|
|
if (__predict_true(!vm_page_wired(m))) {
|
|
|
|
vm_page_free(m);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
vm_page_xunbusy(m);
|
2019-07-29 22:01:28 +00:00
|
|
|
} else {
|
2019-12-28 19:03:32 +00:00
|
|
|
vm_page_release_toq(m, PQ_INACTIVE, flags != 0);
|
2019-07-29 22:01:28 +00:00
|
|
|
}
|
2017-09-24 16:50:10 +00:00
|
|
|
}
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static bool
|
|
|
|
vm_page_try_blocked_op(vm_page_t m, void (*op)(vm_page_t))
|
|
|
|
{
|
|
|
|
u_int old;
|
|
|
|
|
|
|
|
KASSERT(m->object != NULL && (m->oflags & VPO_UNMANAGED) == 0,
|
|
|
|
("vm_page_try_blocked_op: page %p has no object", m));
|
2019-10-15 03:35:11 +00:00
|
|
|
KASSERT(vm_page_busied(m),
|
|
|
|
("vm_page_try_blocked_op: page %p is not busy", m));
|
Change synchonization rules for vm_page reference counting.
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
2019-09-09 21:32:42 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(m->object);
|
|
|
|
|
|
|
|
old = m->ref_count;
|
|
|
|
do {
|
|
|
|
KASSERT(old != 0,
|
|
|
|
("vm_page_try_blocked_op: page %p has no references", m));
|
|
|
|
if (VPRC_WIRE_COUNT(old) != 0)
|
|
|
|
return (false);
|
|
|
|
} while (!atomic_fcmpset_int(&m->ref_count, &old, old | VPRC_BLOCKED));
|
|
|
|
|
|
|
|
(op)(m);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the object is read-locked, new wirings may be created via an
|
|
|
|
* object lookup.
|
|
|
|
*/
|
|
|
|
old = vm_page_drop(m, VPRC_BLOCKED);
|
|
|
|
KASSERT(!VM_OBJECT_WOWNED(m->object) ||
|
|
|
|
old == (VPRC_BLOCKED | VPRC_OBJREF),
|
|
|
|
("vm_page_try_blocked_op: unexpected refcount value %u for %p",
|
|
|
|
old, m));
|
|
|
|
return (true);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Atomically check for wirings and remove all mappings of the page.
|
|
|
|
*/
|
|
|
|
bool
|
|
|
|
vm_page_try_remove_all(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (vm_page_try_blocked_op(m, pmap_remove_all));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Atomically check for wirings and remove all writeable mappings of the page.
|
|
|
|
*/
|
|
|
|
bool
|
|
|
|
vm_page_try_remove_write(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (vm_page_try_blocked_op(m, pmap_remove_write));
|
2001-05-24 07:22:27 +00:00
|
|
|
}
|
|
|
|
|
1999-09-17 04:56:40 +00:00
|
|
|
/*
|
2013-06-10 01:48:21 +00:00
|
|
|
* vm_page_advise
|
1999-09-17 04:56:40 +00:00
|
|
|
*
|
2017-01-15 03:50:08 +00:00
|
|
|
* Apply the specified advice to the given page.
|
1999-09-17 04:56:40 +00:00
|
|
|
*/
|
|
|
|
void
|
2013-06-10 01:48:21 +00:00
|
|
|
vm_page_advise(vm_page_t m, int advice)
|
1999-09-17 04:56:40 +00:00
|
|
|
{
|
|
|
|
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(m->object);
|
2020-08-04 14:59:43 +00:00
|
|
|
vm_page_assert_xbusied(m);
|
|
|
|
|
2015-08-28 00:44:17 +00:00
|
|
|
if (advice == MADV_FREE)
|
2013-06-10 01:48:21 +00:00
|
|
|
/*
|
|
|
|
* Mark the page clean. This will allow the page to be freed
|
2016-12-12 17:47:09 +00:00
|
|
|
* without first paging it out. MADV_FREE pages are often
|
|
|
|
* quickly reused by malloc(3), so we do not do anything that
|
|
|
|
* would result in a page fault on a later access.
|
2013-06-10 01:48:21 +00:00
|
|
|
*/
|
2016-07-29 21:05:37 +00:00
|
|
|
vm_page_undirty(m);
|
2017-01-15 03:50:08 +00:00
|
|
|
else if (advice != MADV_DONTNEED) {
|
|
|
|
if (advice == MADV_WILLNEED)
|
|
|
|
vm_page_activate(m);
|
2013-06-10 01:48:21 +00:00
|
|
|
return;
|
2017-01-15 03:50:08 +00:00
|
|
|
}
|
1999-09-17 04:56:40 +00:00
|
|
|
|
2019-12-28 19:04:00 +00:00
|
|
|
if (advice != MADV_FREE && m->dirty == 0 && pmap_is_modified(m))
|
|
|
|
vm_page_dirty(m);
|
|
|
|
|
Essentially, neither madvise(..., MADV_DONTNEED) nor madvise(..., MADV_FREE)
work. (Moreover, I don't believe that they have ever worked as intended.)
The explanation is fairly simple. Both MADV_DONTNEED and MADV_FREE perform
vm_page_dontneed() on each page within the range given to madvise(). This
function moves the page to the inactive queue. Specifically, if the page is
clean, it is moved to the head of the inactive queue where it is first in
line for processing by the page daemon. On the other hand, if it is dirty,
it is placed at the tail. Let's further examine the case in which the page
is clean. Recall that the page is at the head of the line for processing by
the page daemon. The expectation of vm_page_dontneed()'s author was that
the page would be transferred from the inactive queue to the cache queue by
the page daemon. (Once the page is in the cache queue, it is, in effect,
free, that is, it can be reallocated to a new vm object by vm_page_alloc()
if it isn't reactivated quickly enough by a user of the old vm object.) The
trouble is that nowhere in the execution of either MADV_DONTNEED or
MADV_FREE is either the machine-independent reference flag (PG_REFERENCED)
or the reference bit in any page table entry (PTE) mapping the page cleared.
Consequently, the immediate reaction of the page daemon is to reactivate the
page because it is referenced. In effect, the madvise() was for naught.
The case in which the page was dirty is not too different. Instead of being
laundered, the page is reactivated.
Note: The essential difference between MADV_DONTNEED and MADV_FREE is
that MADV_FREE clears a page's dirty field. So, MADV_FREE is always
executing the clean case above.
This revision changes vm_page_dontneed() to clear both the machine-
independent reference flag (PG_REFERENCED) and the reference bit in all PTEs
mapping the page.
MFC after: 6 weeks
2008-06-06 18:38:43 +00:00
|
|
|
/*
|
|
|
|
* Clear any references to the page. Otherwise, the page daemon will
|
|
|
|
* immediately reactivate the page.
|
|
|
|
*/
|
2011-09-06 10:30:11 +00:00
|
|
|
vm_page_aflag_clear(m, PGA_REFERENCED);
|
Essentially, neither madvise(..., MADV_DONTNEED) nor madvise(..., MADV_FREE)
work. (Moreover, I don't believe that they have ever worked as intended.)
The explanation is fairly simple. Both MADV_DONTNEED and MADV_FREE perform
vm_page_dontneed() on each page within the range given to madvise(). This
function moves the page to the inactive queue. Specifically, if the page is
clean, it is moved to the head of the inactive queue where it is first in
line for processing by the page daemon. On the other hand, if it is dirty,
it is placed at the tail. Let's further examine the case in which the page
is clean. Recall that the page is at the head of the line for processing by
the page daemon. The expectation of vm_page_dontneed()'s author was that
the page would be transferred from the inactive queue to the cache queue by
the page daemon. (Once the page is in the cache queue, it is, in effect,
free, that is, it can be reallocated to a new vm object by vm_page_alloc()
if it isn't reactivated quickly enough by a user of the old vm object.) The
trouble is that nowhere in the execution of either MADV_DONTNEED or
MADV_FREE is either the machine-independent reference flag (PG_REFERENCED)
or the reference bit in any page table entry (PTE) mapping the page cleared.
Consequently, the immediate reaction of the page daemon is to reactivate the
page because it is referenced. In effect, the madvise() was for naught.
The case in which the page was dirty is not too different. Instead of being
laundered, the page is reactivated.
Note: The essential difference between MADV_DONTNEED and MADV_FREE is
that MADV_FREE clears a page's dirty field. So, MADV_FREE is always
executing the clean case above.
This revision changes vm_page_dontneed() to clear both the machine-
independent reference flag (PG_REFERENCED) and the reference bit in all PTEs
mapping the page.
MFC after: 6 weeks
2008-06-06 18:38:43 +00:00
|
|
|
|
2015-08-28 00:44:17 +00:00
|
|
|
/*
|
2016-07-23 21:02:36 +00:00
|
|
|
* Place clean pages near the head of the inactive queue rather than
|
|
|
|
* the tail, thus defeating the queue's LRU operation and ensuring that
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
* the page will be reused quickly. Dirty pages not already in the
|
|
|
|
* laundry are moved there.
|
2015-08-28 00:44:17 +00:00
|
|
|
*/
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
if (m->dirty == 0)
|
|
|
|
vm_page_deactivate_noreuse(m);
|
2018-03-18 16:40:56 +00:00
|
|
|
else if (!vm_page_in_laundry(m))
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
vm_page_launder(m);
|
1999-09-17 04:56:40 +00:00
|
|
|
}
|
|
|
|
|
2020-02-27 02:37:27 +00:00
|
|
|
/*
|
|
|
|
* vm_page_grab_release
|
|
|
|
*
|
|
|
|
* Helper routine for grab functions to release busy on return.
|
|
|
|
*/
|
|
|
|
static inline void
|
|
|
|
vm_page_grab_release(vm_page_t m, int allocflags)
|
|
|
|
{
|
|
|
|
|
|
|
|
if ((allocflags & VM_ALLOC_NOBUSY) != 0) {
|
|
|
|
if ((allocflags & VM_ALLOC_IGN_SBUSY) != 0)
|
|
|
|
vm_page_sunbusy(m);
|
|
|
|
else
|
|
|
|
vm_page_xunbusy(m);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_grab_sleep
|
|
|
|
*
|
|
|
|
* Sleep for busy according to VM_ALLOC_ parameters. Returns true
|
|
|
|
* if the caller should retry and false otherwise.
|
|
|
|
*
|
|
|
|
* If the object is locked on entry the object will be unlocked with
|
|
|
|
* false returns and still locked but possibly having been dropped
|
|
|
|
* with true returns.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
vm_page_grab_sleep(vm_object_t object, vm_page_t m, vm_pindex_t pindex,
|
|
|
|
const char *wmesg, int allocflags, bool locked)
|
|
|
|
{
|
|
|
|
|
|
|
|
if ((allocflags & VM_ALLOC_NOWAIT) != 0)
|
|
|
|
return (false);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reference the page before unlocking and sleeping so that
|
|
|
|
* the page daemon is less likely to reclaim it.
|
|
|
|
*/
|
|
|
|
if (locked && (allocflags & VM_ALLOC_NOCREAT) == 0)
|
|
|
|
vm_page_reference(m);
|
|
|
|
|
2020-12-27 21:50:54 +00:00
|
|
|
if (_vm_page_busy_sleep(object, m, pindex, wmesg, allocflags, locked) &&
|
|
|
|
locked)
|
2020-02-27 02:37:27 +00:00
|
|
|
VM_OBJECT_WLOCK(object);
|
|
|
|
if ((allocflags & VM_ALLOC_WAITFAIL) != 0)
|
|
|
|
return (false);
|
|
|
|
|
|
|
|
return (true);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Assert that the grab flags are valid.
|
|
|
|
*/
|
|
|
|
static inline void
|
|
|
|
vm_page_grab_check(int allocflags)
|
2019-12-08 01:16:22 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
KASSERT((allocflags & VM_ALLOC_NOBUSY) == 0 ||
|
|
|
|
(allocflags & VM_ALLOC_WIRED) != 0,
|
2020-02-27 02:37:27 +00:00
|
|
|
("vm_page_grab*: the pages must be busied or wired"));
|
|
|
|
|
2019-12-08 01:16:22 +00:00
|
|
|
KASSERT((allocflags & VM_ALLOC_SBUSY) == 0 ||
|
|
|
|
(allocflags & VM_ALLOC_IGN_SBUSY) != 0,
|
2020-02-27 02:37:27 +00:00
|
|
|
("vm_page_grab*: VM_ALLOC_SBUSY/VM_ALLOC_IGN_SBUSY mismatch"));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Calculate the page allocation flags for grab.
|
|
|
|
*/
|
|
|
|
static inline int
|
|
|
|
vm_page_grab_pflags(int allocflags)
|
|
|
|
{
|
|
|
|
int pflags;
|
|
|
|
|
2019-12-08 01:16:22 +00:00
|
|
|
pflags = allocflags &
|
|
|
|
~(VM_ALLOC_NOWAIT | VM_ALLOC_WAITOK | VM_ALLOC_WAITFAIL |
|
2021-10-20 00:25:14 +00:00
|
|
|
VM_ALLOC_NOBUSY | VM_ALLOC_IGN_SBUSY);
|
2019-12-08 01:16:22 +00:00
|
|
|
if ((allocflags & VM_ALLOC_NOWAIT) == 0)
|
|
|
|
pflags |= VM_ALLOC_WAITFAIL;
|
|
|
|
if ((allocflags & VM_ALLOC_IGN_SBUSY) != 0)
|
|
|
|
pflags |= VM_ALLOC_SBUSY;
|
|
|
|
|
|
|
|
return (pflags);
|
|
|
|
}
|
|
|
|
|
1998-02-05 03:32:49 +00:00
|
|
|
/*
|
|
|
|
* Grab a page, waiting until we are waken up due to the page
|
|
|
|
* changing state. We keep on waiting, if the page continues
|
2004-04-24 21:36:23 +00:00
|
|
|
* to be in the object. If the page doesn't exist, first allocate it
|
|
|
|
* and then conditionally zero it.
|
1998-12-23 01:52:47 +00:00
|
|
|
*
|
2012-10-03 05:06:45 +00:00
|
|
|
* This routine may sleep.
|
|
|
|
*
|
|
|
|
* The object must be locked on entry. The lock will, however, be released
|
|
|
|
* and reacquired if the routine sleeps.
|
1998-02-05 03:32:49 +00:00
|
|
|
*/
|
|
|
|
vm_page_t
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_grab(vm_object_t object, vm_pindex_t pindex, int allocflags)
|
1998-02-05 03:32:49 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
2020-02-27 02:37:27 +00:00
|
|
|
vm_page_grab_check(allocflags);
|
|
|
|
|
1998-02-05 03:32:49 +00:00
|
|
|
retrylookup:
|
|
|
|
if ((m = vm_page_lookup(object, pindex)) != NULL) {
|
2020-02-27 02:37:27 +00:00
|
|
|
if (!vm_page_tryacquire(m, allocflags)) {
|
|
|
|
if (vm_page_grab_sleep(object, m, pindex, "pgrbwt",
|
|
|
|
allocflags, true))
|
2019-12-08 01:16:22 +00:00
|
|
|
goto retrylookup;
|
|
|
|
return (NULL);
|
1998-02-05 03:32:49 +00:00
|
|
|
}
|
2019-12-08 01:16:22 +00:00
|
|
|
goto out;
|
1998-02-05 03:32:49 +00:00
|
|
|
}
|
2019-09-10 19:08:01 +00:00
|
|
|
if ((allocflags & VM_ALLOC_NOCREAT) != 0)
|
|
|
|
return (NULL);
|
2020-02-27 02:37:27 +00:00
|
|
|
m = vm_page_alloc(object, pindex, vm_page_grab_pflags(allocflags));
|
1998-02-05 03:32:49 +00:00
|
|
|
if (m == NULL) {
|
2020-02-13 23:18:35 +00:00
|
|
|
if ((allocflags & (VM_ALLOC_NOWAIT | VM_ALLOC_WAITFAIL)) != 0)
|
2014-12-22 09:02:21 +00:00
|
|
|
return (NULL);
|
1998-02-05 03:32:49 +00:00
|
|
|
goto retrylookup;
|
2016-11-15 18:22:50 +00:00
|
|
|
}
|
2004-04-24 20:53:55 +00:00
|
|
|
if (allocflags & VM_ALLOC_ZERO && (m->flags & PG_ZERO) == 0)
|
|
|
|
pmap_zero_page(m);
|
2019-10-15 03:35:11 +00:00
|
|
|
|
|
|
|
out:
|
2020-02-27 02:37:27 +00:00
|
|
|
vm_page_grab_release(m, allocflags);
|
|
|
|
|
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Locklessly attempt to acquire a page given a (object, pindex) tuple
|
|
|
|
* and an optional previous page to avoid the radix lookup. The resulting
|
|
|
|
* page will be validated against the identity tuple and busied or wired
|
|
|
|
* as requested. A NULL *mp return guarantees that the page was not in
|
|
|
|
* radix at the time of the call but callers must perform higher level
|
|
|
|
* synchronization or retry the operation under a lock if they require
|
|
|
|
* an atomic answer. This is the only lock free validation routine,
|
|
|
|
* other routines can depend on the resulting page state.
|
|
|
|
*
|
|
|
|
* The return value indicates whether the operation failed due to caller
|
|
|
|
* flags. The return is tri-state with mp:
|
|
|
|
*
|
|
|
|
* (true, *mp != NULL) - The operation was successful.
|
|
|
|
* (true, *mp == NULL) - The page was not found in tree.
|
|
|
|
* (false, *mp == NULL) - WAITFAIL or NOWAIT prevented acquisition.
|
|
|
|
*/
|
|
|
|
static bool
|
|
|
|
vm_page_acquire_unlocked(vm_object_t object, vm_pindex_t pindex,
|
|
|
|
vm_page_t prev, vm_page_t *mp, int allocflags)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
|
|
|
vm_page_grab_check(allocflags);
|
|
|
|
MPASS(prev == NULL || vm_page_busied(prev) || vm_page_wired(prev));
|
|
|
|
|
|
|
|
*mp = NULL;
|
|
|
|
for (;;) {
|
|
|
|
/*
|
|
|
|
* We may see a false NULL here because the previous page
|
|
|
|
* has been removed or just inserted and the list is loaded
|
|
|
|
* without barriers. Switch to radix to verify.
|
|
|
|
*/
|
|
|
|
if (prev == NULL || (m = TAILQ_NEXT(prev, listq)) == NULL ||
|
2020-04-20 14:45:17 +00:00
|
|
|
QMD_IS_TRASHED(m) || m->pindex != pindex ||
|
2020-02-27 02:37:27 +00:00
|
|
|
atomic_load_ptr(&m->object) != object) {
|
|
|
|
prev = NULL;
|
|
|
|
/*
|
|
|
|
* This guarantees the result is instantaneously
|
|
|
|
* correct.
|
|
|
|
*/
|
|
|
|
m = vm_radix_lookup_unlocked(&object->rtree, pindex);
|
|
|
|
}
|
|
|
|
if (m == NULL)
|
|
|
|
return (true);
|
|
|
|
if (vm_page_trybusy(m, allocflags)) {
|
|
|
|
if (m->object == object && m->pindex == pindex)
|
|
|
|
break;
|
|
|
|
/* relookup. */
|
|
|
|
vm_page_busy_release(m);
|
|
|
|
cpu_spinwait();
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
if (!vm_page_grab_sleep(object, m, pindex, "pgnslp",
|
|
|
|
allocflags, false))
|
|
|
|
return (false);
|
2019-10-15 03:35:11 +00:00
|
|
|
}
|
2020-02-27 02:37:27 +00:00
|
|
|
if ((allocflags & VM_ALLOC_WIRED) != 0)
|
|
|
|
vm_page_wire(m);
|
|
|
|
vm_page_grab_release(m, allocflags);
|
|
|
|
*mp = m;
|
|
|
|
return (true);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Try to locklessly grab a page and fall back to the object lock if NOCREAT
|
|
|
|
* is not set.
|
|
|
|
*/
|
|
|
|
vm_page_t
|
|
|
|
vm_page_grab_unlocked(vm_object_t object, vm_pindex_t pindex, int allocflags)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
|
|
|
vm_page_grab_check(allocflags);
|
|
|
|
|
|
|
|
if (!vm_page_acquire_unlocked(object, pindex, NULL, &m, allocflags))
|
|
|
|
return (NULL);
|
|
|
|
if (m != NULL)
|
|
|
|
return (m);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The radix lockless lookup should never return a false negative
|
|
|
|
* errors. If the user specifies NOCREAT they are guaranteed there
|
|
|
|
* was no page present at the instant of the call. A NOCREAT caller
|
|
|
|
* must handle create races gracefully.
|
|
|
|
*/
|
|
|
|
if ((allocflags & VM_ALLOC_NOCREAT) != 0)
|
|
|
|
return (NULL);
|
|
|
|
|
|
|
|
VM_OBJECT_WLOCK(object);
|
|
|
|
m = vm_page_grab(object, pindex, allocflags);
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
|
2004-04-24 21:36:23 +00:00
|
|
|
return (m);
|
1998-02-05 03:32:49 +00:00
|
|
|
}
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2019-09-10 19:08:01 +00:00
|
|
|
/*
|
|
|
|
* Grab a page and make it valid, paging in if necessary. Pages missing from
|
2019-12-15 02:00:32 +00:00
|
|
|
* their pager are zero filled and validated. If a VM_ALLOC_COUNT is supplied
|
|
|
|
* and the page is not valid as many as VM_INITIAL_PAGEIN pages can be brought
|
|
|
|
* in simultaneously. Additional pages will be left on a paging queue but
|
|
|
|
* will neither be wired nor busy regardless of allocflags.
|
2019-09-10 19:08:01 +00:00
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_grab_valid(vm_page_t *mp, vm_object_t object, vm_pindex_t pindex, int allocflags)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
2019-12-15 02:00:32 +00:00
|
|
|
vm_page_t ma[VM_INITIAL_PAGEIN];
|
|
|
|
int after, i, pflags, rv;
|
2019-09-10 19:08:01 +00:00
|
|
|
|
|
|
|
KASSERT((allocflags & VM_ALLOC_SBUSY) == 0 ||
|
|
|
|
(allocflags & VM_ALLOC_IGN_SBUSY) != 0,
|
|
|
|
("vm_page_grab_valid: VM_ALLOC_SBUSY/VM_ALLOC_IGN_SBUSY mismatch"));
|
|
|
|
KASSERT((allocflags &
|
|
|
|
(VM_ALLOC_NOWAIT | VM_ALLOC_WAITFAIL | VM_ALLOC_ZERO)) == 0,
|
|
|
|
("vm_page_grab_valid: Invalid flags 0x%X", allocflags));
|
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
2020-03-01 22:09:11 +00:00
|
|
|
pflags = allocflags & ~(VM_ALLOC_NOBUSY | VM_ALLOC_SBUSY |
|
2021-10-20 00:25:14 +00:00
|
|
|
VM_ALLOC_WIRED | VM_ALLOC_IGN_SBUSY);
|
2019-09-10 19:08:01 +00:00
|
|
|
pflags |= VM_ALLOC_WAITFAIL;
|
|
|
|
|
|
|
|
retrylookup:
|
|
|
|
if ((m = vm_page_lookup(object, pindex)) != NULL) {
|
|
|
|
/*
|
|
|
|
* If the page is fully valid it can only become invalid
|
|
|
|
* with the object lock held. If it is not valid it can
|
|
|
|
* become valid with the busy lock held. Therefore, we
|
|
|
|
* may unnecessarily lock the exclusive busy here if we
|
|
|
|
* race with I/O completion not using the object lock.
|
|
|
|
* However, we will not end up with an invalid page and a
|
|
|
|
* shared lock.
|
|
|
|
*/
|
2020-02-27 02:37:27 +00:00
|
|
|
if (!vm_page_trybusy(m,
|
|
|
|
vm_page_all_valid(m) ? allocflags : 0)) {
|
|
|
|
(void)vm_page_grab_sleep(object, m, pindex, "pgrbwt",
|
|
|
|
allocflags, true);
|
2019-09-10 19:08:01 +00:00
|
|
|
goto retrylookup;
|
|
|
|
}
|
2020-02-27 02:37:27 +00:00
|
|
|
if (vm_page_all_valid(m))
|
|
|
|
goto out;
|
|
|
|
if ((allocflags & VM_ALLOC_NOCREAT) != 0) {
|
|
|
|
vm_page_busy_release(m);
|
2019-09-10 19:08:01 +00:00
|
|
|
*mp = NULL;
|
|
|
|
return (VM_PAGER_FAIL);
|
|
|
|
}
|
|
|
|
} else if ((allocflags & VM_ALLOC_NOCREAT) != 0) {
|
|
|
|
*mp = NULL;
|
|
|
|
return (VM_PAGER_FAIL);
|
2020-02-27 02:37:27 +00:00
|
|
|
} else if ((m = vm_page_alloc(object, pindex, pflags)) == NULL) {
|
2022-12-17 00:12:14 +00:00
|
|
|
if (!vm_pager_can_alloc_page(object, pindex)) {
|
|
|
|
*mp = NULL;
|
2022-10-20 13:15:46 +00:00
|
|
|
return (VM_PAGER_AGAIN);
|
2022-12-17 00:12:14 +00:00
|
|
|
}
|
2019-09-10 19:08:01 +00:00
|
|
|
goto retrylookup;
|
|
|
|
}
|
|
|
|
|
|
|
|
vm_page_assert_xbusied(m);
|
2019-12-15 02:00:32 +00:00
|
|
|
if (vm_pager_has_page(object, pindex, NULL, &after)) {
|
|
|
|
after = MIN(after, VM_INITIAL_PAGEIN);
|
|
|
|
after = MIN(after, allocflags >> VM_ALLOC_COUNT_SHIFT);
|
|
|
|
after = MAX(after, 1);
|
|
|
|
ma[0] = m;
|
|
|
|
for (i = 1; i < after; i++) {
|
|
|
|
if ((ma[i] = vm_page_next(ma[i - 1])) != NULL) {
|
2022-10-18 09:16:36 +00:00
|
|
|
if (vm_page_any_valid(ma[i]) ||
|
|
|
|
!vm_page_tryxbusy(ma[i]))
|
2019-12-15 02:00:32 +00:00
|
|
|
break;
|
|
|
|
} else {
|
|
|
|
ma[i] = vm_page_alloc(object, m->pindex + i,
|
|
|
|
VM_ALLOC_NORMAL);
|
|
|
|
if (ma[i] == NULL)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
after = i;
|
2020-01-19 23:47:32 +00:00
|
|
|
vm_object_pip_add(object, after);
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
2019-12-15 02:00:32 +00:00
|
|
|
rv = vm_pager_get_pages(object, ma, after, NULL, NULL);
|
2020-01-19 23:47:32 +00:00
|
|
|
VM_OBJECT_WLOCK(object);
|
|
|
|
vm_object_pip_wakeupn(object, after);
|
2019-12-15 02:00:32 +00:00
|
|
|
/* Pager may have replaced a page. */
|
|
|
|
m = ma[0];
|
2019-09-10 19:08:01 +00:00
|
|
|
if (rv != VM_PAGER_OK) {
|
2019-12-15 02:00:32 +00:00
|
|
|
for (i = 0; i < after; i++) {
|
|
|
|
if (!vm_page_wired(ma[i]))
|
|
|
|
vm_page_free(ma[i]);
|
|
|
|
else
|
|
|
|
vm_page_xunbusy(ma[i]);
|
|
|
|
}
|
2019-09-10 19:08:01 +00:00
|
|
|
*mp = NULL;
|
|
|
|
return (rv);
|
|
|
|
}
|
2019-12-15 02:00:32 +00:00
|
|
|
for (i = 1; i < after; i++)
|
|
|
|
vm_page_readahead_finish(ma[i]);
|
2019-10-15 03:45:41 +00:00
|
|
|
MPASS(vm_page_all_valid(m));
|
2019-09-10 19:08:01 +00:00
|
|
|
} else {
|
|
|
|
vm_page_zero_invalid(m, TRUE);
|
|
|
|
}
|
|
|
|
out:
|
2020-02-27 02:37:27 +00:00
|
|
|
if ((allocflags & VM_ALLOC_WIRED) != 0)
|
|
|
|
vm_page_wire(m);
|
|
|
|
if ((allocflags & VM_ALLOC_SBUSY) != 0 && vm_page_xbusied(m))
|
2019-09-10 19:08:01 +00:00
|
|
|
vm_page_busy_downgrade(m);
|
2020-02-27 02:37:27 +00:00
|
|
|
else if ((allocflags & VM_ALLOC_NOBUSY) != 0)
|
|
|
|
vm_page_busy_release(m);
|
2019-09-10 19:08:01 +00:00
|
|
|
*mp = m;
|
|
|
|
return (VM_PAGER_OK);
|
|
|
|
}
|
|
|
|
|
2020-02-27 02:37:27 +00:00
|
|
|
/*
|
|
|
|
* Locklessly grab a valid page. If the page is not valid or not yet
|
|
|
|
* allocated this will fall back to the object lock method.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_grab_valid_unlocked(vm_page_t *mp, vm_object_t object,
|
|
|
|
vm_pindex_t pindex, int allocflags)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
int flags;
|
|
|
|
int error;
|
|
|
|
|
|
|
|
KASSERT((allocflags & VM_ALLOC_SBUSY) == 0 ||
|
|
|
|
(allocflags & VM_ALLOC_IGN_SBUSY) != 0,
|
|
|
|
("vm_page_grab_valid_unlocked: VM_ALLOC_SBUSY/VM_ALLOC_IGN_SBUSY "
|
|
|
|
"mismatch"));
|
|
|
|
KASSERT((allocflags &
|
|
|
|
(VM_ALLOC_NOWAIT | VM_ALLOC_WAITFAIL | VM_ALLOC_ZERO)) == 0,
|
|
|
|
("vm_page_grab_valid_unlocked: Invalid flags 0x%X", allocflags));
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Attempt a lockless lookup and busy. We need at least an sbusy
|
|
|
|
* before we can inspect the valid field and return a wired page.
|
|
|
|
*/
|
|
|
|
flags = allocflags & ~(VM_ALLOC_NOBUSY | VM_ALLOC_WIRED);
|
|
|
|
if (!vm_page_acquire_unlocked(object, pindex, NULL, mp, flags))
|
|
|
|
return (VM_PAGER_FAIL);
|
|
|
|
if ((m = *mp) != NULL) {
|
|
|
|
if (vm_page_all_valid(m)) {
|
|
|
|
if ((allocflags & VM_ALLOC_WIRED) != 0)
|
|
|
|
vm_page_wire(m);
|
|
|
|
vm_page_grab_release(m, allocflags);
|
|
|
|
return (VM_PAGER_OK);
|
|
|
|
}
|
|
|
|
vm_page_busy_release(m);
|
|
|
|
}
|
2020-02-28 20:32:35 +00:00
|
|
|
if ((allocflags & VM_ALLOC_NOCREAT) != 0) {
|
|
|
|
*mp = NULL;
|
|
|
|
return (VM_PAGER_FAIL);
|
|
|
|
}
|
2020-02-27 02:37:27 +00:00
|
|
|
VM_OBJECT_WLOCK(object);
|
|
|
|
error = vm_page_grab_valid(mp, object, pindex, allocflags);
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
|
|
|
|
return (error);
|
|
|
|
}
|
|
|
|
|
2017-08-09 04:23:04 +00:00
|
|
|
/*
|
|
|
|
* Return the specified range of pages from the given object. For each
|
|
|
|
* page offset within the range, if a page already exists within the object
|
|
|
|
* at that offset and it is busy, then wait for it to change state. If,
|
|
|
|
* instead, the page doesn't exist, then allocate it.
|
|
|
|
*
|
|
|
|
* The caller must always specify an allocation class.
|
|
|
|
*
|
|
|
|
* allocation classes:
|
|
|
|
* VM_ALLOC_NORMAL normal process request
|
|
|
|
* VM_ALLOC_SYSTEM system *really* needs the pages
|
|
|
|
*
|
|
|
|
* The caller must always specify that the pages are to be busied and/or
|
|
|
|
* wired.
|
|
|
|
*
|
|
|
|
* optional allocation flags:
|
|
|
|
* VM_ALLOC_IGN_SBUSY do not sleep on soft busy pages
|
|
|
|
* VM_ALLOC_NOBUSY do not exclusive busy the page
|
2017-08-11 16:29:22 +00:00
|
|
|
* VM_ALLOC_NOWAIT do not sleep
|
2017-08-09 04:23:04 +00:00
|
|
|
* VM_ALLOC_SBUSY set page to sbusy state
|
|
|
|
* VM_ALLOC_WIRED wire the pages
|
|
|
|
* VM_ALLOC_ZERO zero and validate any invalid pages
|
|
|
|
*
|
2017-08-11 16:29:22 +00:00
|
|
|
* If VM_ALLOC_NOWAIT is not specified, this routine may sleep. Otherwise, it
|
|
|
|
* may return a partial prefix of the requested range.
|
2017-08-09 04:23:04 +00:00
|
|
|
*/
|
2017-08-11 16:29:22 +00:00
|
|
|
int
|
2017-08-09 04:23:04 +00:00
|
|
|
vm_page_grab_pages(vm_object_t object, vm_pindex_t pindex, int allocflags,
|
|
|
|
vm_page_t *ma, int count)
|
|
|
|
{
|
2017-08-15 16:39:49 +00:00
|
|
|
vm_page_t m, mpred;
|
2017-11-08 02:39:37 +00:00
|
|
|
int pflags;
|
2017-08-09 04:23:04 +00:00
|
|
|
int i;
|
|
|
|
|
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
|
|
|
KASSERT(((u_int)allocflags >> VM_ALLOC_COUNT_SHIFT) == 0,
|
|
|
|
("vm_page_grap_pages: VM_ALLOC_COUNT() is not allowed"));
|
2020-09-02 19:59:25 +00:00
|
|
|
KASSERT(count > 0,
|
|
|
|
("vm_page_grab_pages: invalid page count %d", count));
|
2020-02-27 02:37:27 +00:00
|
|
|
vm_page_grab_check(allocflags);
|
2019-12-08 01:16:22 +00:00
|
|
|
|
|
|
|
pflags = vm_page_grab_pflags(allocflags);
|
2017-08-09 04:23:04 +00:00
|
|
|
i = 0;
|
|
|
|
retrylookup:
|
2017-08-15 16:39:49 +00:00
|
|
|
m = vm_radix_lookup_le(&object->rtree, pindex + i);
|
|
|
|
if (m == NULL || m->pindex != pindex + i) {
|
|
|
|
mpred = m;
|
|
|
|
m = NULL;
|
|
|
|
} else
|
|
|
|
mpred = TAILQ_PREV(m, pglist, listq);
|
2017-08-09 04:23:04 +00:00
|
|
|
for (; i < count; i++) {
|
|
|
|
if (m != NULL) {
|
2020-02-27 02:37:27 +00:00
|
|
|
if (!vm_page_tryacquire(m, allocflags)) {
|
2020-12-27 21:50:54 +00:00
|
|
|
if (vm_page_grab_sleep(object, m, pindex + i,
|
2020-02-27 02:37:27 +00:00
|
|
|
"grbmaw", allocflags, true))
|
2019-12-08 01:16:22 +00:00
|
|
|
goto retrylookup;
|
|
|
|
break;
|
2017-08-09 04:23:04 +00:00
|
|
|
}
|
|
|
|
} else {
|
2019-09-10 19:08:01 +00:00
|
|
|
if ((allocflags & VM_ALLOC_NOCREAT) != 0)
|
|
|
|
break;
|
2017-08-15 16:39:49 +00:00
|
|
|
m = vm_page_alloc_after(object, pindex + i,
|
2017-11-08 02:39:37 +00:00
|
|
|
pflags | VM_ALLOC_COUNT(count - i), mpred);
|
2017-08-09 04:23:04 +00:00
|
|
|
if (m == NULL) {
|
2020-02-13 23:18:35 +00:00
|
|
|
if ((allocflags & (VM_ALLOC_NOWAIT |
|
|
|
|
VM_ALLOC_WAITFAIL)) != 0)
|
2017-08-11 16:29:22 +00:00
|
|
|
break;
|
2017-08-09 04:23:04 +00:00
|
|
|
goto retrylookup;
|
|
|
|
}
|
|
|
|
}
|
2019-10-15 03:45:41 +00:00
|
|
|
if (vm_page_none_valid(m) &&
|
|
|
|
(allocflags & VM_ALLOC_ZERO) != 0) {
|
2017-08-09 04:23:04 +00:00
|
|
|
if ((m->flags & PG_ZERO) == 0)
|
|
|
|
pmap_zero_page(m);
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_valid(m);
|
2017-08-09 04:23:04 +00:00
|
|
|
}
|
2020-02-27 02:37:27 +00:00
|
|
|
vm_page_grab_release(m, allocflags);
|
2017-08-15 16:39:49 +00:00
|
|
|
ma[i] = mpred = m;
|
2017-08-09 04:23:04 +00:00
|
|
|
m = vm_page_next(m);
|
|
|
|
}
|
2017-08-11 16:29:22 +00:00
|
|
|
return (i);
|
2017-08-09 04:23:04 +00:00
|
|
|
}
|
|
|
|
|
2020-02-27 02:37:27 +00:00
|
|
|
/*
|
|
|
|
* Unlocked variant of vm_page_grab_pages(). This accepts the same flags
|
|
|
|
* and will fall back to the locked variant to handle allocation.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_grab_pages_unlocked(vm_object_t object, vm_pindex_t pindex,
|
|
|
|
int allocflags, vm_page_t *ma, int count)
|
|
|
|
{
|
|
|
|
vm_page_t m, pred;
|
|
|
|
int flags;
|
|
|
|
int i;
|
|
|
|
|
2020-09-02 19:59:25 +00:00
|
|
|
KASSERT(count > 0,
|
|
|
|
("vm_page_grab_pages_unlocked: invalid page count %d", count));
|
2020-02-27 02:37:27 +00:00
|
|
|
vm_page_grab_check(allocflags);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Modify flags for lockless acquire to hold the page until we
|
|
|
|
* set it valid if necessary.
|
|
|
|
*/
|
|
|
|
flags = allocflags & ~VM_ALLOC_NOBUSY;
|
|
|
|
pred = NULL;
|
|
|
|
for (i = 0; i < count; i++, pindex++) {
|
|
|
|
if (!vm_page_acquire_unlocked(object, pindex, pred, &m, flags))
|
|
|
|
return (i);
|
|
|
|
if (m == NULL)
|
|
|
|
break;
|
|
|
|
if ((flags & VM_ALLOC_ZERO) != 0 && vm_page_none_valid(m)) {
|
|
|
|
if ((m->flags & PG_ZERO) == 0)
|
|
|
|
pmap_zero_page(m);
|
|
|
|
vm_page_valid(m);
|
|
|
|
}
|
|
|
|
/* m will still be wired or busy according to flags. */
|
|
|
|
vm_page_grab_release(m, allocflags);
|
|
|
|
pred = ma[i] = m;
|
|
|
|
}
|
2020-09-02 19:59:25 +00:00
|
|
|
if (i == count || (allocflags & VM_ALLOC_NOCREAT) != 0)
|
2020-02-27 02:37:27 +00:00
|
|
|
return (i);
|
|
|
|
count -= i;
|
|
|
|
VM_OBJECT_WLOCK(object);
|
|
|
|
i += vm_page_grab_pages(object, pindex, allocflags, &ma[i], count);
|
|
|
|
VM_OBJECT_WUNLOCK(object);
|
|
|
|
|
|
|
|
return (i);
|
|
|
|
}
|
|
|
|
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
/*
|
2012-10-03 05:06:45 +00:00
|
|
|
* Mapping function for valid or dirty bits in a page.
|
1999-04-05 19:38:30 +00:00
|
|
|
*
|
|
|
|
* Inputs are required to range within a page.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*/
|
2011-11-05 08:20:32 +00:00
|
|
|
vm_page_bits_t
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
vm_page_bits(int base, int size)
|
|
|
|
{
|
1999-04-05 19:38:30 +00:00
|
|
|
int first_bit;
|
|
|
|
int last_bit;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1999-04-05 19:38:30 +00:00
|
|
|
KASSERT(
|
|
|
|
base + size <= PAGE_SIZE,
|
|
|
|
("vm_page_bits: illegal base/size %d/%d", base, size)
|
|
|
|
);
|
Some VM improvements, including elimination of alot of Sig-11
problems. Tor Egge and others have helped with various VM bugs
lately, but don't blame him -- blame me!!!
pmap.c:
1) Create an object for kernel page table allocations. This
fixes a bogus allocation method previously used for such, by
grabbing pages from the kernel object, using bogus pindexes.
(This was a code cleanup, and perhaps a minor system stability
issue.)
pmap.c:
2) Pre-set the modify and accessed bits when prudent. This will
decrease bus traffic under certain circumstances.
vfs_bio.c, vfs_cluster.c:
3) Rather than calculating the beginning virtual byte offset
multiple times, stick the offset into the buffer header, so
that the calculated offset can be reused. (Long long multiplies
are often expensive, and this is a probably unmeasurable performance
improvement, and code cleanup.)
vfs_bio.c:
4) Handle write recursion more intelligently (but not perfectly) so
that it is less likely to cause a system panic, and is also
much more robust.
vfs_bio.c:
5) getblk incorrectly wrote out blocks that are incorrectly sized.
The problem is fixed, and writes blocks out ONLY when B_DELWRI
is true.
vfs_bio.c:
6) Check that already constituted buffers have fully valid pages. If
not, then make sure that the B_CACHE bit is not set. (This was
a major source of Sig-11 type problems.)
vfs_bio.c:
7) Fix a potential system deadlock due to an incorrectly specified
sleep priority while waiting for a buffer write operation. The
change that I made opens the system up to serious problems, and
we need to examine the issue of process sleep priorities.
vfs_cluster.c, vfs_bio.c:
8) Make clustered reads work more correctly (and more completely)
when buffers are already constituted, but not fully valid.
(This was another system reliability issue.)
vfs_subr.c, ffs_inode.c:
9) Create a vtruncbuf function, which is used by filesystems that
can truncate files. The vinvalbuf forced a file sync type operation,
while vtruncbuf only invalidates the buffers past the new end of file,
and also invalidates the appropriate pages. (This was a system reliabiliy
and performance issue.)
10) Modify FFS to use vtruncbuf.
vm_object.c:
11) Make the object rundown mechanism for OBJT_VNODE type objects work
more correctly. Included in that fix, create pager entries for
the OBJT_DEAD pager type, so that paging requests that might slip
in during race conditions are properly handled. (This was a system
reliability issue.)
vm_page.c:
12) Make some of the page validation routines be a little less picky
about arguments passed to them. Also, support page invalidation
change the object generation count so that we handle generation
counts a little more robustly.
vm_pageout.c:
13) Further reduce pageout daemon activity when the system doesn't
need help from it. There should be no additional performance
decrease even when the pageout daemon is running. (This was
a significant performance issue.)
vnode_pager.c:
14) Teach the vnode pager to handle race conditions during vnode
deallocations.
1998-03-16 01:56:03 +00:00
|
|
|
|
1999-04-05 19:38:30 +00:00
|
|
|
if (size == 0) /* handle degenerate case */
|
2002-03-10 21:52:48 +00:00
|
|
|
return (0);
|
Some VM improvements, including elimination of alot of Sig-11
problems. Tor Egge and others have helped with various VM bugs
lately, but don't blame him -- blame me!!!
pmap.c:
1) Create an object for kernel page table allocations. This
fixes a bogus allocation method previously used for such, by
grabbing pages from the kernel object, using bogus pindexes.
(This was a code cleanup, and perhaps a minor system stability
issue.)
pmap.c:
2) Pre-set the modify and accessed bits when prudent. This will
decrease bus traffic under certain circumstances.
vfs_bio.c, vfs_cluster.c:
3) Rather than calculating the beginning virtual byte offset
multiple times, stick the offset into the buffer header, so
that the calculated offset can be reused. (Long long multiplies
are often expensive, and this is a probably unmeasurable performance
improvement, and code cleanup.)
vfs_bio.c:
4) Handle write recursion more intelligently (but not perfectly) so
that it is less likely to cause a system panic, and is also
much more robust.
vfs_bio.c:
5) getblk incorrectly wrote out blocks that are incorrectly sized.
The problem is fixed, and writes blocks out ONLY when B_DELWRI
is true.
vfs_bio.c:
6) Check that already constituted buffers have fully valid pages. If
not, then make sure that the B_CACHE bit is not set. (This was
a major source of Sig-11 type problems.)
vfs_bio.c:
7) Fix a potential system deadlock due to an incorrectly specified
sleep priority while waiting for a buffer write operation. The
change that I made opens the system up to serious problems, and
we need to examine the issue of process sleep priorities.
vfs_cluster.c, vfs_bio.c:
8) Make clustered reads work more correctly (and more completely)
when buffers are already constituted, but not fully valid.
(This was another system reliability issue.)
vfs_subr.c, ffs_inode.c:
9) Create a vtruncbuf function, which is used by filesystems that
can truncate files. The vinvalbuf forced a file sync type operation,
while vtruncbuf only invalidates the buffers past the new end of file,
and also invalidates the appropriate pages. (This was a system reliabiliy
and performance issue.)
10) Modify FFS to use vtruncbuf.
vm_object.c:
11) Make the object rundown mechanism for OBJT_VNODE type objects work
more correctly. Included in that fix, create pager entries for
the OBJT_DEAD pager type, so that paging requests that might slip
in during race conditions are properly handled. (This was a system
reliability issue.)
vm_page.c:
12) Make some of the page validation routines be a little less picky
about arguments passed to them. Also, support page invalidation
change the object generation count so that we handle generation
counts a little more robustly.
vm_pageout.c:
13) Further reduce pageout daemon activity when the system doesn't
need help from it. There should be no additional performance
decrease even when the pageout daemon is running. (This was
a significant performance issue.)
vnode_pager.c:
14) Teach the vnode pager to handle race conditions during vnode
deallocations.
1998-03-16 01:56:03 +00:00
|
|
|
|
1999-04-05 19:38:30 +00:00
|
|
|
first_bit = base >> DEV_BSHIFT;
|
|
|
|
last_bit = (base + size - 1) >> DEV_BSHIFT;
|
|
|
|
|
2011-11-05 08:20:32 +00:00
|
|
|
return (((vm_page_bits_t)2 << last_bit) -
|
|
|
|
((vm_page_bits_t)1 << first_bit));
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
|
|
|
|
2019-11-19 23:30:09 +00:00
|
|
|
void
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_bits_set(vm_page_t m, vm_page_bits_t *bits, vm_page_bits_t set)
|
|
|
|
{
|
|
|
|
|
|
|
|
#if PAGE_SIZE == 32768
|
|
|
|
atomic_set_64((uint64_t *)bits, set);
|
|
|
|
#elif PAGE_SIZE == 16384
|
|
|
|
atomic_set_32((uint32_t *)bits, set);
|
|
|
|
#elif (PAGE_SIZE == 8192) && defined(atomic_set_16)
|
|
|
|
atomic_set_16((uint16_t *)bits, set);
|
|
|
|
#elif (PAGE_SIZE == 4096) && defined(atomic_set_8)
|
|
|
|
atomic_set_8((uint8_t *)bits, set);
|
|
|
|
#else /* PAGE_SIZE <= 8192 */
|
|
|
|
uintptr_t addr;
|
|
|
|
int shift;
|
|
|
|
|
|
|
|
addr = (uintptr_t)bits;
|
|
|
|
/*
|
|
|
|
* Use a trick to perform a 32-bit atomic on the
|
|
|
|
* containing aligned word, to not depend on the existence
|
|
|
|
* of atomic_{set, clear}_{8, 16}.
|
|
|
|
*/
|
|
|
|
shift = addr & (sizeof(uint32_t) - 1);
|
|
|
|
#if BYTE_ORDER == BIG_ENDIAN
|
|
|
|
shift = (sizeof(uint32_t) - sizeof(vm_page_bits_t) - shift) * NBBY;
|
|
|
|
#else
|
|
|
|
shift *= NBBY;
|
|
|
|
#endif
|
|
|
|
addr &= ~(sizeof(uint32_t) - 1);
|
|
|
|
atomic_set_32((uint32_t *)addr, set << shift);
|
|
|
|
#endif /* PAGE_SIZE */
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void
|
|
|
|
vm_page_bits_clear(vm_page_t m, vm_page_bits_t *bits, vm_page_bits_t clear)
|
|
|
|
{
|
|
|
|
|
|
|
|
#if PAGE_SIZE == 32768
|
|
|
|
atomic_clear_64((uint64_t *)bits, clear);
|
|
|
|
#elif PAGE_SIZE == 16384
|
|
|
|
atomic_clear_32((uint32_t *)bits, clear);
|
|
|
|
#elif (PAGE_SIZE == 8192) && defined(atomic_clear_16)
|
|
|
|
atomic_clear_16((uint16_t *)bits, clear);
|
|
|
|
#elif (PAGE_SIZE == 4096) && defined(atomic_clear_8)
|
|
|
|
atomic_clear_8((uint8_t *)bits, clear);
|
|
|
|
#else /* PAGE_SIZE <= 8192 */
|
|
|
|
uintptr_t addr;
|
|
|
|
int shift;
|
|
|
|
|
|
|
|
addr = (uintptr_t)bits;
|
|
|
|
/*
|
|
|
|
* Use a trick to perform a 32-bit atomic on the
|
|
|
|
* containing aligned word, to not depend on the existence
|
|
|
|
* of atomic_{set, clear}_{8, 16}.
|
|
|
|
*/
|
|
|
|
shift = addr & (sizeof(uint32_t) - 1);
|
|
|
|
#if BYTE_ORDER == BIG_ENDIAN
|
|
|
|
shift = (sizeof(uint32_t) - sizeof(vm_page_bits_t) - shift) * NBBY;
|
|
|
|
#else
|
|
|
|
shift *= NBBY;
|
|
|
|
#endif
|
|
|
|
addr &= ~(sizeof(uint32_t) - 1);
|
|
|
|
atomic_clear_32((uint32_t *)addr, clear << shift);
|
|
|
|
#endif /* PAGE_SIZE */
|
|
|
|
}
|
|
|
|
|
2019-12-15 03:15:06 +00:00
|
|
|
static inline vm_page_bits_t
|
|
|
|
vm_page_bits_swap(vm_page_t m, vm_page_bits_t *bits, vm_page_bits_t newbits)
|
|
|
|
{
|
|
|
|
#if PAGE_SIZE == 32768
|
|
|
|
uint64_t old;
|
|
|
|
|
|
|
|
old = *bits;
|
|
|
|
while (atomic_fcmpset_64(bits, &old, newbits) == 0);
|
|
|
|
return (old);
|
|
|
|
#elif PAGE_SIZE == 16384
|
|
|
|
uint32_t old;
|
|
|
|
|
|
|
|
old = *bits;
|
|
|
|
while (atomic_fcmpset_32(bits, &old, newbits) == 0);
|
|
|
|
return (old);
|
|
|
|
#elif (PAGE_SIZE == 8192) && defined(atomic_fcmpset_16)
|
|
|
|
uint16_t old;
|
|
|
|
|
|
|
|
old = *bits;
|
|
|
|
while (atomic_fcmpset_16(bits, &old, newbits) == 0);
|
|
|
|
return (old);
|
|
|
|
#elif (PAGE_SIZE == 4096) && defined(atomic_fcmpset_8)
|
|
|
|
uint8_t old;
|
|
|
|
|
|
|
|
old = *bits;
|
|
|
|
while (atomic_fcmpset_8(bits, &old, newbits) == 0);
|
|
|
|
return (old);
|
|
|
|
#else /* PAGE_SIZE <= 4096*/
|
|
|
|
uintptr_t addr;
|
|
|
|
uint32_t old, new, mask;
|
|
|
|
int shift;
|
|
|
|
|
|
|
|
addr = (uintptr_t)bits;
|
|
|
|
/*
|
|
|
|
* Use a trick to perform a 32-bit atomic on the
|
|
|
|
* containing aligned word, to not depend on the existence
|
|
|
|
* of atomic_{set, swap, clear}_{8, 16}.
|
|
|
|
*/
|
|
|
|
shift = addr & (sizeof(uint32_t) - 1);
|
|
|
|
#if BYTE_ORDER == BIG_ENDIAN
|
|
|
|
shift = (sizeof(uint32_t) - sizeof(vm_page_bits_t) - shift) * NBBY;
|
|
|
|
#else
|
|
|
|
shift *= NBBY;
|
|
|
|
#endif
|
|
|
|
addr &= ~(sizeof(uint32_t) - 1);
|
|
|
|
mask = VM_PAGE_BITS_ALL << shift;
|
|
|
|
|
|
|
|
old = *bits;
|
|
|
|
do {
|
|
|
|
new = old & ~mask;
|
|
|
|
new |= newbits << shift;
|
|
|
|
} while (atomic_fcmpset_32((uint32_t *)addr, &old, new) == 0);
|
|
|
|
return (old >> shift);
|
|
|
|
#endif /* PAGE_SIZE */
|
|
|
|
}
|
|
|
|
|
2009-05-13 05:39:39 +00:00
|
|
|
/*
|
2011-11-30 17:39:00 +00:00
|
|
|
* vm_page_set_valid_range:
|
2009-05-13 05:39:39 +00:00
|
|
|
*
|
|
|
|
* Sets portions of a page valid. The arguments are expected
|
|
|
|
* to be DEV_BSIZE aligned but if they aren't the bitmap is inclusive
|
|
|
|
* of any partial chunks touched by the range. The invalid portion of
|
|
|
|
* such chunks will be zeroed.
|
|
|
|
*
|
|
|
|
* (base + size) must be less then or equal to PAGE_SIZE.
|
|
|
|
*/
|
|
|
|
void
|
2011-11-30 17:39:00 +00:00
|
|
|
vm_page_set_valid_range(vm_page_t m, int base, int size)
|
2009-05-13 05:39:39 +00:00
|
|
|
{
|
|
|
|
int endoff, frag;
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_bits_t pagebits;
|
2009-05-13 05:39:39 +00:00
|
|
|
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_assert_busied(m);
|
2009-05-13 05:39:39 +00:00
|
|
|
if (size == 0) /* handle degenerate case */
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the base is not DEV_BSIZE aligned and the valid
|
|
|
|
* bit is clear, we have to zero out a portion of the
|
|
|
|
* first block.
|
|
|
|
*/
|
2016-04-21 19:57:40 +00:00
|
|
|
if ((frag = rounddown2(base, DEV_BSIZE)) != base &&
|
2009-05-13 05:39:39 +00:00
|
|
|
(m->valid & (1 << (base >> DEV_BSHIFT))) == 0)
|
|
|
|
pmap_zero_page_area(m, frag, base - frag);
|
|
|
|
|
|
|
|
/*
|
2015-03-24 20:07:27 +00:00
|
|
|
* If the ending offset is not DEV_BSIZE aligned and the
|
2009-05-13 05:39:39 +00:00
|
|
|
* valid bit is clear, we have to zero out a portion of
|
|
|
|
* the last block.
|
|
|
|
*/
|
|
|
|
endoff = base + size;
|
2016-04-21 19:57:40 +00:00
|
|
|
if ((frag = rounddown2(endoff, DEV_BSIZE)) != endoff &&
|
2009-05-13 05:39:39 +00:00
|
|
|
(m->valid & (1 << (endoff >> DEV_BSHIFT))) == 0)
|
|
|
|
pmap_zero_page_area(m, endoff,
|
|
|
|
DEV_BSIZE - (endoff & (DEV_BSIZE - 1)));
|
|
|
|
|
2009-05-30 22:06:58 +00:00
|
|
|
/*
|
|
|
|
* Assert that no previously invalid block that is now being validated
|
2015-03-24 20:07:27 +00:00
|
|
|
* is already dirty.
|
2009-05-30 22:06:58 +00:00
|
|
|
*/
|
|
|
|
KASSERT((~m->valid & vm_page_bits(base, size) & m->dirty) == 0,
|
2011-11-30 17:39:00 +00:00
|
|
|
("vm_page_set_valid_range: page %p is dirty", m));
|
2009-05-30 22:06:58 +00:00
|
|
|
|
2009-05-13 05:39:39 +00:00
|
|
|
/*
|
|
|
|
* Set valid bits inclusive of any overlap.
|
|
|
|
*/
|
2019-10-15 03:45:41 +00:00
|
|
|
pagebits = vm_page_bits(base, size);
|
|
|
|
if (vm_page_xbusied(m))
|
|
|
|
m->valid |= pagebits;
|
|
|
|
else
|
|
|
|
vm_page_bits_set(m, &m->valid, pagebits);
|
2009-05-13 05:39:39 +00:00
|
|
|
}
|
|
|
|
|
2019-12-15 03:15:06 +00:00
|
|
|
/*
|
|
|
|
* Set the page dirty bits and free the invalid swap space if
|
|
|
|
* present. Returns the previous dirty bits.
|
|
|
|
*/
|
|
|
|
vm_page_bits_t
|
|
|
|
vm_page_set_dirty(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_page_bits_t old;
|
|
|
|
|
|
|
|
VM_PAGE_OBJECT_BUSY_ASSERT(m);
|
|
|
|
|
|
|
|
if (vm_page_xbusied(m) && !pmap_page_is_write_mapped(m)) {
|
|
|
|
old = m->dirty;
|
|
|
|
m->dirty = VM_PAGE_BITS_ALL;
|
|
|
|
} else
|
|
|
|
old = vm_page_bits_swap(m, &m->dirty, VM_PAGE_BITS_ALL);
|
|
|
|
if (old == 0 && (m->a.flags & PGA_SWAP_SPACE) != 0)
|
|
|
|
vm_pager_page_unswapped(m);
|
|
|
|
|
|
|
|
return (old);
|
|
|
|
}
|
|
|
|
|
2010-06-02 15:46:37 +00:00
|
|
|
/*
|
|
|
|
* Clear the given bits from the specified page's dirty field.
|
|
|
|
*/
|
|
|
|
static __inline void
|
2011-11-05 08:20:32 +00:00
|
|
|
vm_page_clear_dirty_mask(vm_page_t m, vm_page_bits_t pagebits)
|
2010-06-02 15:46:37 +00:00
|
|
|
{
|
2019-10-15 03:45:41 +00:00
|
|
|
|
|
|
|
vm_page_assert_busied(m);
|
2010-06-02 15:46:37 +00:00
|
|
|
|
|
|
|
/*
|
2019-10-15 03:45:41 +00:00
|
|
|
* If the page is xbusied and not write mapped we are the
|
|
|
|
* only thread that can modify dirty bits. Otherwise, The pmap
|
|
|
|
* layer can call vm_page_dirty() without holding a distinguished
|
|
|
|
* lock. The combination of page busy and atomic operations
|
|
|
|
* suffice to guarantee consistency of the page dirty field.
|
2010-06-02 15:46:37 +00:00
|
|
|
*/
|
2019-10-15 03:45:41 +00:00
|
|
|
if (vm_page_xbusied(m) && !pmap_page_is_write_mapped(m))
|
2010-06-02 15:46:37 +00:00
|
|
|
m->dirty &= ~pagebits;
|
2019-10-15 03:45:41 +00:00
|
|
|
else
|
|
|
|
vm_page_bits_clear(m, &m->dirty, pagebits);
|
2010-06-02 15:46:37 +00:00
|
|
|
}
|
|
|
|
|
1995-09-03 19:57:25 +00:00
|
|
|
/*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* vm_page_set_validclean:
|
1999-04-05 19:38:30 +00:00
|
|
|
*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* Sets portions of a page valid and clean. The arguments are expected
|
|
|
|
* to be DEV_BSIZE aligned but if they aren't the bitmap is inclusive
|
|
|
|
* of any partial chunks touched by the range. The invalid portion of
|
|
|
|
* such chunks will be zero'd.
|
1999-04-05 19:38:30 +00:00
|
|
|
*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* (base + size) must be less then or equal to PAGE_SIZE.
|
1995-09-03 19:57:25 +00:00
|
|
|
*/
|
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_set_validclean(vm_page_t m, int base, int size)
|
1995-09-03 19:57:25 +00:00
|
|
|
{
|
2011-11-05 08:20:32 +00:00
|
|
|
vm_page_bits_t oldvalid, pagebits;
|
|
|
|
int endoff, frag;
|
1999-04-05 19:38:30 +00:00
|
|
|
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_assert_busied(m);
|
1999-04-05 19:38:30 +00:00
|
|
|
if (size == 0) /* handle degenerate case */
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the base is not DEV_BSIZE aligned and the valid
|
|
|
|
* bit is clear, we have to zero out a portion of the
|
|
|
|
* first block.
|
|
|
|
*/
|
2016-04-21 19:57:40 +00:00
|
|
|
if ((frag = rounddown2(base, DEV_BSIZE)) != base &&
|
2011-11-05 08:20:32 +00:00
|
|
|
(m->valid & ((vm_page_bits_t)1 << (base >> DEV_BSHIFT))) == 0)
|
2002-04-15 16:00:03 +00:00
|
|
|
pmap_zero_page_area(m, frag, base - frag);
|
1999-04-05 19:38:30 +00:00
|
|
|
|
|
|
|
/*
|
2015-03-24 20:07:27 +00:00
|
|
|
* If the ending offset is not DEV_BSIZE aligned and the
|
1999-04-05 19:38:30 +00:00
|
|
|
* valid bit is clear, we have to zero out a portion of
|
|
|
|
* the last block.
|
|
|
|
*/
|
|
|
|
endoff = base + size;
|
2016-04-21 19:57:40 +00:00
|
|
|
if ((frag = rounddown2(endoff, DEV_BSIZE)) != endoff &&
|
2011-11-05 08:20:32 +00:00
|
|
|
(m->valid & ((vm_page_bits_t)1 << (endoff >> DEV_BSHIFT))) == 0)
|
2002-04-15 16:00:03 +00:00
|
|
|
pmap_zero_page_area(m, endoff,
|
|
|
|
DEV_BSIZE - (endoff & (DEV_BSIZE - 1)));
|
1999-04-05 19:38:30 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Set valid, clear dirty bits. If validating the entire
|
1999-12-12 03:19:33 +00:00
|
|
|
* page we can safely clear the pmap modify bit. We also
|
2019-10-15 03:48:22 +00:00
|
|
|
* use this opportunity to clear the PGA_NOSYNC flag. If a process
|
1999-12-12 03:19:33 +00:00
|
|
|
* takes a write fault on a MAP_NOSYNC memory area the flag will
|
|
|
|
* be set again.
|
2001-12-14 01:16:57 +00:00
|
|
|
*
|
|
|
|
* We set valid bits inclusive of any overlap, but we can only
|
|
|
|
* clear dirty bits for DEV_BSIZE chunks that are fully within
|
|
|
|
* the range.
|
1999-04-05 19:38:30 +00:00
|
|
|
*/
|
2010-06-02 15:46:37 +00:00
|
|
|
oldvalid = m->valid;
|
1999-04-05 19:38:30 +00:00
|
|
|
pagebits = vm_page_bits(base, size);
|
2019-10-15 03:45:41 +00:00
|
|
|
if (vm_page_xbusied(m))
|
|
|
|
m->valid |= pagebits;
|
|
|
|
else
|
|
|
|
vm_page_bits_set(m, &m->valid, pagebits);
|
2001-12-14 01:16:57 +00:00
|
|
|
#if 0 /* NOT YET */
|
|
|
|
if ((frag = base & (DEV_BSIZE - 1)) != 0) {
|
|
|
|
frag = DEV_BSIZE - frag;
|
|
|
|
base += frag;
|
|
|
|
size -= frag;
|
|
|
|
if (size < 0)
|
|
|
|
size = 0;
|
|
|
|
}
|
|
|
|
pagebits = vm_page_bits(base, size & (DEV_BSIZE - 1));
|
|
|
|
#endif
|
1999-12-12 03:19:33 +00:00
|
|
|
if (base == 0 && size == PAGE_SIZE) {
|
2010-06-02 15:46:37 +00:00
|
|
|
/*
|
|
|
|
* The page can only be modified within the pmap if it is
|
|
|
|
* mapped, and it can only be mapped if it was previously
|
|
|
|
* fully valid.
|
|
|
|
*/
|
|
|
|
if (oldvalid == VM_PAGE_BITS_ALL)
|
|
|
|
/*
|
|
|
|
* Perform the pmap_clear_modify() first. Otherwise,
|
|
|
|
* a concurrent pmap operation, such as
|
|
|
|
* pmap_protect(), could clear a modification in the
|
|
|
|
* pmap and set the dirty field on the page before
|
|
|
|
* pmap_clear_modify() had begun and after the dirty
|
|
|
|
* field was cleared here.
|
|
|
|
*/
|
|
|
|
pmap_clear_modify(m);
|
|
|
|
m->dirty = 0;
|
2019-10-15 03:48:22 +00:00
|
|
|
vm_page_aflag_clear(m, PGA_NOSYNC);
|
2019-10-15 03:45:41 +00:00
|
|
|
} else if (oldvalid != VM_PAGE_BITS_ALL && vm_page_xbusied(m))
|
2010-06-02 15:46:37 +00:00
|
|
|
m->dirty &= ~pagebits;
|
|
|
|
else
|
|
|
|
vm_page_clear_dirty_mask(m, pagebits);
|
1995-09-03 19:57:25 +00:00
|
|
|
}
|
|
|
|
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_clear_dirty(vm_page_t m, int base, int size)
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
{
|
2003-08-23 18:11:53 +00:00
|
|
|
|
2010-06-02 15:46:37 +00:00
|
|
|
vm_page_clear_dirty_mask(m, vm_page_bits(base, size));
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
}
|
|
|
|
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
/*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* vm_page_set_invalid:
|
|
|
|
*
|
|
|
|
* Invalidates DEV_BSIZE'd chunks within a page. Both the
|
|
|
|
* valid and dirty bits for the effected areas are cleared.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*/
|
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_set_invalid(vm_page_t m, int base, int size)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
2011-11-05 08:20:32 +00:00
|
|
|
vm_page_bits_t bits;
|
2013-09-14 10:11:38 +00:00
|
|
|
vm_object_t object;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2019-10-15 03:45:41 +00:00
|
|
|
/*
|
|
|
|
* The object lock is required so that pages can't be mapped
|
|
|
|
* read-only while we're in the process of invalidating them.
|
|
|
|
*/
|
2013-09-14 10:11:38 +00:00
|
|
|
object = m->object;
|
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_assert_busied(m);
|
|
|
|
|
2013-09-14 10:11:38 +00:00
|
|
|
if (object->type == OBJT_VNODE && base == 0 && IDX_TO_OFF(m->pindex) +
|
|
|
|
size >= object->un_pager.vnp.vnp_size)
|
|
|
|
bits = VM_PAGE_BITS_ALL;
|
|
|
|
else
|
|
|
|
bits = vm_page_bits(base, size);
|
2019-10-15 03:45:41 +00:00
|
|
|
if (object->ref_count != 0 && vm_page_all_valid(m) && bits != 0)
|
2006-01-24 07:21:38 +00:00
|
|
|
pmap_remove_all(m);
|
2019-10-15 03:45:41 +00:00
|
|
|
KASSERT((bits == 0 && vm_page_all_valid(m)) ||
|
2013-09-14 10:11:38 +00:00
|
|
|
!pmap_page_is_mapped(m),
|
2010-05-18 16:40:29 +00:00
|
|
|
("vm_page_set_invalid: page %p is mapped", m));
|
2019-10-15 03:45:41 +00:00
|
|
|
if (vm_page_xbusied(m)) {
|
|
|
|
m->valid &= ~bits;
|
|
|
|
m->dirty &= ~bits;
|
|
|
|
} else {
|
|
|
|
vm_page_bits_clear(m, &m->valid, bits);
|
|
|
|
vm_page_bits_clear(m, &m->dirty, bits);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_invalid:
|
|
|
|
*
|
|
|
|
* Invalidates the entire page. The page must be busy, unmapped, and
|
|
|
|
* the enclosing object must be locked. The object locks protects
|
|
|
|
* against concurrent read-only pmap enter which is done without
|
|
|
|
* busy.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_invalid(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
|
|
|
vm_page_assert_busied(m);
|
2021-12-05 15:39:58 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(m->object);
|
2019-10-15 03:45:41 +00:00
|
|
|
MPASS(!pmap_page_is_mapped(m));
|
|
|
|
|
|
|
|
if (vm_page_xbusied(m))
|
|
|
|
m->valid = 0;
|
|
|
|
else
|
|
|
|
vm_page_bits_clear(m, &m->valid, VM_PAGE_BITS_ALL);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1999-04-05 19:38:30 +00:00
|
|
|
* vm_page_zero_invalid()
|
|
|
|
*
|
2015-03-24 20:07:27 +00:00
|
|
|
* The kernel assumes that the invalid portions of a page contain
|
1999-04-05 19:38:30 +00:00
|
|
|
* garbage, but such pages can be mapped into memory by user code.
|
|
|
|
* When this occurs, we must zero out the non-valid portions of the
|
|
|
|
* page so user code sees what it expects.
|
|
|
|
*
|
2015-03-24 20:07:27 +00:00
|
|
|
* Pages are most often semi-valid when the end of a file is mapped
|
1999-04-05 19:38:30 +00:00
|
|
|
* into memory and the file's size is not page aligned.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_zero_invalid(vm_page_t m, boolean_t setvalid)
|
|
|
|
{
|
|
|
|
int b;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Scan the valid bits looking for invalid sections that
|
2015-05-20 23:15:56 +00:00
|
|
|
* must be zeroed. Invalid sub-DEV_BSIZE'd areas ( where the
|
|
|
|
* valid bit may be set ) have already been zeroed by
|
1999-04-05 19:38:30 +00:00
|
|
|
* vm_page_set_validclean().
|
|
|
|
*/
|
|
|
|
for (b = i = 0; i <= PAGE_SIZE / DEV_BSIZE; ++i) {
|
2015-03-24 20:07:27 +00:00
|
|
|
if (i == (PAGE_SIZE / DEV_BSIZE) ||
|
2011-11-05 08:20:32 +00:00
|
|
|
(m->valid & ((vm_page_bits_t)1 << i))) {
|
1999-04-05 19:38:30 +00:00
|
|
|
if (i > b) {
|
2015-03-24 20:07:27 +00:00
|
|
|
pmap_zero_page_area(m,
|
2002-04-15 16:00:03 +00:00
|
|
|
b << DEV_BSHIFT, (i - b) << DEV_BSHIFT);
|
1999-04-05 19:38:30 +00:00
|
|
|
}
|
|
|
|
b = i + 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* setvalid is TRUE when we can safely set the zero'd areas
|
2022-06-04 10:52:22 +00:00
|
|
|
* as being valid. We can do this if there are no cache consistency
|
1999-04-05 19:38:30 +00:00
|
|
|
* issues. e.g. it is ok to do with UFS, but not ok to do with NFS.
|
|
|
|
*/
|
|
|
|
if (setvalid)
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_valid(m);
|
1999-04-05 19:38:30 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_is_valid:
|
|
|
|
*
|
|
|
|
* Is (partial) page valid? Note that the case where size == 0
|
|
|
|
* will return FALSE in the degenerate case where the page is
|
|
|
|
* entirely invalid, and TRUE otherwise.
|
2019-10-15 03:45:41 +00:00
|
|
|
*
|
|
|
|
* Some callers envoke this routine without the busy lock held and
|
|
|
|
* handle races via higher level locks. Typical callers should
|
|
|
|
* hold a busy lock to prevent invalidation.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*/
|
|
|
|
int
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_is_valid(vm_page_t m, int base, int size)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
2011-11-05 08:20:32 +00:00
|
|
|
vm_page_bits_t bits;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2011-11-05 08:20:32 +00:00
|
|
|
bits = vm_page_bits(base, size);
|
2022-10-18 09:16:36 +00:00
|
|
|
return (vm_page_any_valid(m) && (m->valid & bits) == bits);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
|
|
|
|
Add a page size field to struct vm_page. Increase the page size field when
a partially populated reservation becomes fully populated, and decrease this
field when a fully populated reservation becomes partially populated.
Use this field to simplify the implementation of pmap_enter_object() on
amd64, arm, and i386.
On all architectures where we support superpages, the cost of creating a
superpage mapping is roughly the same as creating a base page mapping. For
example, both kinds of mappings entail the creation of a single PTE and PV
entry. With this in mind, use the page size field to make the
implementation of vm_map_pmap_enter(..., MAP_PREFAULT_PARTIAL) a little
smarter. Previously, if MAP_PREFAULT_PARTIAL was specified to
vm_map_pmap_enter(), that function would only map base pages. Now, it will
create up to 96 base page or superpage mappings.
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
2014-06-07 17:12:26 +00:00
|
|
|
/*
|
2017-07-14 02:15:48 +00:00
|
|
|
* Returns true if all of the specified predicates are true for the entire
|
|
|
|
* (super)page and false otherwise.
|
Add a page size field to struct vm_page. Increase the page size field when
a partially populated reservation becomes fully populated, and decrease this
field when a fully populated reservation becomes partially populated.
Use this field to simplify the implementation of pmap_enter_object() on
amd64, arm, and i386.
On all architectures where we support superpages, the cost of creating a
superpage mapping is roughly the same as creating a base page mapping. For
example, both kinds of mappings entail the creation of a single PTE and PV
entry. With this in mind, use the page size field to make the
implementation of vm_map_pmap_enter(..., MAP_PREFAULT_PARTIAL) a little
smarter. Previously, if MAP_PREFAULT_PARTIAL was specified to
vm_map_pmap_enter(), that function would only map base pages. Now, it will
create up to 96 base page or superpage mappings.
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
2014-06-07 17:12:26 +00:00
|
|
|
*/
|
2017-07-14 02:15:48 +00:00
|
|
|
bool
|
|
|
|
vm_page_ps_test(vm_page_t m, int flags, vm_page_t skip_m)
|
Add a page size field to struct vm_page. Increase the page size field when
a partially populated reservation becomes fully populated, and decrease this
field when a fully populated reservation becomes partially populated.
Use this field to simplify the implementation of pmap_enter_object() on
amd64, arm, and i386.
On all architectures where we support superpages, the cost of creating a
superpage mapping is roughly the same as creating a base page mapping. For
example, both kinds of mappings entail the creation of a single PTE and PV
entry. With this in mind, use the page size field to make the
implementation of vm_map_pmap_enter(..., MAP_PREFAULT_PARTIAL) a little
smarter. Previously, if MAP_PREFAULT_PARTIAL was specified to
vm_map_pmap_enter(), that function would only map base pages. Now, it will
create up to 96 base page or superpage mappings.
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
2014-06-07 17:12:26 +00:00
|
|
|
{
|
2017-07-23 05:54:56 +00:00
|
|
|
vm_object_t object;
|
Add a page size field to struct vm_page. Increase the page size field when
a partially populated reservation becomes fully populated, and decrease this
field when a fully populated reservation becomes partially populated.
Use this field to simplify the implementation of pmap_enter_object() on
amd64, arm, and i386.
On all architectures where we support superpages, the cost of creating a
superpage mapping is roughly the same as creating a base page mapping. For
example, both kinds of mappings entail the creation of a single PTE and PV
entry. With this in mind, use the page size field to make the
implementation of vm_map_pmap_enter(..., MAP_PREFAULT_PARTIAL) a little
smarter. Previously, if MAP_PREFAULT_PARTIAL was specified to
vm_map_pmap_enter(), that function would only map base pages. Now, it will
create up to 96 base page or superpage mappings.
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
2014-06-07 17:12:26 +00:00
|
|
|
int i, npages;
|
|
|
|
|
2017-07-23 05:54:56 +00:00
|
|
|
object = m->object;
|
2018-04-17 18:49:17 +00:00
|
|
|
if (skip_m != NULL && skip_m->object != object)
|
|
|
|
return (false);
|
2017-07-23 05:54:56 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(object);
|
Add a page size field to struct vm_page. Increase the page size field when
a partially populated reservation becomes fully populated, and decrease this
field when a fully populated reservation becomes partially populated.
Use this field to simplify the implementation of pmap_enter_object() on
amd64, arm, and i386.
On all architectures where we support superpages, the cost of creating a
superpage mapping is roughly the same as creating a base page mapping. For
example, both kinds of mappings entail the creation of a single PTE and PV
entry. With this in mind, use the page size field to make the
implementation of vm_map_pmap_enter(..., MAP_PREFAULT_PARTIAL) a little
smarter. Previously, if MAP_PREFAULT_PARTIAL was specified to
vm_map_pmap_enter(), that function would only map base pages. Now, it will
create up to 96 base page or superpage mappings.
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
2014-06-07 17:12:26 +00:00
|
|
|
npages = atop(pagesizes[m->psind]);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The physically contiguous pages that make up a superpage, i.e., a
|
|
|
|
* page with a page size index ("psind") greater than zero, will
|
|
|
|
* occupy adjacent entries in vm_page_array[].
|
|
|
|
*/
|
|
|
|
for (i = 0; i < npages; i++) {
|
2017-07-23 05:54:56 +00:00
|
|
|
/* Always test object consistency, including "skip_m". */
|
|
|
|
if (m[i].object != object)
|
|
|
|
return (false);
|
2017-07-14 02:15:48 +00:00
|
|
|
if (&m[i] == skip_m)
|
|
|
|
continue;
|
|
|
|
if ((flags & PS_NONE_BUSY) != 0 && vm_page_busied(&m[i]))
|
|
|
|
return (false);
|
|
|
|
if ((flags & PS_ALL_DIRTY) != 0) {
|
|
|
|
/*
|
|
|
|
* Calling vm_page_test_dirty() or pmap_is_modified()
|
|
|
|
* might stop this case from spuriously returning
|
|
|
|
* "false". However, that would require a write lock
|
|
|
|
* on the object containing "m[i]".
|
|
|
|
*/
|
|
|
|
if (m[i].dirty != VM_PAGE_BITS_ALL)
|
|
|
|
return (false);
|
|
|
|
}
|
|
|
|
if ((flags & PS_ALL_VALID) != 0 &&
|
|
|
|
m[i].valid != VM_PAGE_BITS_ALL)
|
|
|
|
return (false);
|
Add a page size field to struct vm_page. Increase the page size field when
a partially populated reservation becomes fully populated, and decrease this
field when a fully populated reservation becomes partially populated.
Use this field to simplify the implementation of pmap_enter_object() on
amd64, arm, and i386.
On all architectures where we support superpages, the cost of creating a
superpage mapping is roughly the same as creating a base page mapping. For
example, both kinds of mappings entail the creation of a single PTE and PV
entry. With this in mind, use the page size field to make the
implementation of vm_map_pmap_enter(..., MAP_PREFAULT_PARTIAL) a little
smarter. Previously, if MAP_PREFAULT_PARTIAL was specified to
vm_map_pmap_enter(), that function would only map base pages. Now, it will
create up to 96 base page or superpage mappings.
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
2014-06-07 17:12:26 +00:00
|
|
|
}
|
2017-07-14 02:15:48 +00:00
|
|
|
return (true);
|
Add a page size field to struct vm_page. Increase the page size field when
a partially populated reservation becomes fully populated, and decrease this
field when a fully populated reservation becomes partially populated.
Use this field to simplify the implementation of pmap_enter_object() on
amd64, arm, and i386.
On all architectures where we support superpages, the cost of creating a
superpage mapping is roughly the same as creating a base page mapping. For
example, both kinds of mappings entail the creation of a single PTE and PV
entry. With this in mind, use the page size field to make the
implementation of vm_map_pmap_enter(..., MAP_PREFAULT_PARTIAL) a little
smarter. Previously, if MAP_PREFAULT_PARTIAL was specified to
vm_map_pmap_enter(), that function would only map base pages. Now, it will
create up to 96 base page or superpage mappings.
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
2014-06-07 17:12:26 +00:00
|
|
|
}
|
|
|
|
|
1998-12-23 01:52:47 +00:00
|
|
|
/*
|
2012-10-03 05:06:45 +00:00
|
|
|
* Set the page's dirty bits if the page is modified.
|
1998-12-23 01:52:47 +00:00
|
|
|
*/
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_test_dirty(vm_page_t m)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
2010-05-18 16:40:29 +00:00
|
|
|
|
2019-10-15 03:45:41 +00:00
|
|
|
vm_page_assert_busied(m);
|
2010-05-18 16:40:29 +00:00
|
|
|
if (m->dirty != VM_PAGE_BITS_ALL && pmap_is_modified(m))
|
1999-01-24 06:00:31 +00:00
|
|
|
vm_page_dirty(m);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
|
|
|
|
2019-10-15 03:45:41 +00:00
|
|
|
void
|
|
|
|
vm_page_valid(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
|
|
|
vm_page_assert_busied(m);
|
|
|
|
if (vm_page_xbusied(m))
|
|
|
|
m->valid = VM_PAGE_BITS_ALL;
|
|
|
|
else
|
|
|
|
vm_page_bits_set(m, &m->valid, VM_PAGE_BITS_ALL);
|
|
|
|
}
|
|
|
|
|
2011-11-29 13:07:32 +00:00
|
|
|
void
|
|
|
|
vm_page_lock_KBI(vm_page_t m, const char *file, int line)
|
|
|
|
{
|
|
|
|
|
|
|
|
mtx_lock_flags_(vm_page_lockptr(m), 0, file, line);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_unlock_KBI(vm_page_t m, const char *file, int line)
|
|
|
|
{
|
|
|
|
|
|
|
|
mtx_unlock_flags_(vm_page_lockptr(m), 0, file, line);
|
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
vm_page_trylock_KBI(vm_page_t m, const char *file, int line)
|
|
|
|
{
|
|
|
|
|
|
|
|
return (mtx_trylock_flags_(vm_page_lockptr(m), 0, file, line));
|
|
|
|
}
|
|
|
|
|
|
|
|
#if defined(INVARIANTS) || defined(INVARIANT_SUPPORT)
|
2013-06-03 01:22:54 +00:00
|
|
|
void
|
|
|
|
vm_page_assert_locked_KBI(vm_page_t m, const char *file, int line)
|
|
|
|
{
|
|
|
|
|
|
|
|
vm_page_lock_assert_KBI(m, MA_OWNED, file, line);
|
|
|
|
}
|
|
|
|
|
2011-11-29 13:07:32 +00:00
|
|
|
void
|
|
|
|
vm_page_lock_assert_KBI(vm_page_t m, int a, const char *file, int line)
|
|
|
|
{
|
|
|
|
|
|
|
|
mtx_assert_(vm_page_lockptr(m), a, file, line);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2011-06-11 20:15:19 +00:00
|
|
|
#ifdef INVARIANTS
|
|
|
|
void
|
2019-10-15 03:41:36 +00:00
|
|
|
vm_page_object_busy_assert(vm_page_t m)
|
2011-06-11 20:15:19 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Certain of the page's fields may only be modified by the
|
2019-10-15 03:41:36 +00:00
|
|
|
* holder of a page or object busy.
|
2011-06-11 20:15:19 +00:00
|
|
|
*/
|
2019-10-15 03:41:36 +00:00
|
|
|
if (m->object != NULL && !vm_page_busied(m))
|
|
|
|
VM_OBJECT_ASSERT_BUSY(m->object);
|
2011-06-11 20:15:19 +00:00
|
|
|
}
|
2014-08-09 05:00:34 +00:00
|
|
|
|
|
|
|
void
|
2019-11-18 18:22:41 +00:00
|
|
|
vm_page_assert_pga_writeable(vm_page_t m, uint16_t bits)
|
2014-08-09 05:00:34 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
if ((bits & PGA_WRITEABLE) == 0)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The PGA_WRITEABLE flag can only be set if the page is
|
|
|
|
* managed, is exclusively busied or the object is locked.
|
|
|
|
* Currently, this flag is only set by pmap_enter().
|
|
|
|
*/
|
|
|
|
KASSERT((m->oflags & VPO_UNMANAGED) == 0,
|
|
|
|
("PGA_WRITEABLE on unmanaged page"));
|
|
|
|
if (!vm_page_xbusied(m))
|
2019-10-15 03:41:36 +00:00
|
|
|
VM_OBJECT_ASSERT_BUSY(m->object);
|
2014-08-09 05:00:34 +00:00
|
|
|
}
|
2011-06-11 20:15:19 +00:00
|
|
|
#endif
|
|
|
|
|
1996-09-14 11:54:59 +00:00
|
|
|
#include "opt_ddb.h"
|
1995-04-16 09:59:16 +00:00
|
|
|
#ifdef DDB
|
1996-09-14 11:54:59 +00:00
|
|
|
#include <sys/kernel.h>
|
|
|
|
|
|
|
|
#include <ddb/ddb.h>
|
|
|
|
|
2022-07-18 21:21:38 +00:00
|
|
|
DB_SHOW_COMMAND_FLAGS(page, vm_page_print_page_info, DB_CMD_MEMSAFE)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
2016-11-22 18:13:46 +00:00
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
db_printf("vm_cnt.v_free_count: %d\n", vm_free_count());
|
|
|
|
db_printf("vm_cnt.v_inactive_count: %d\n", vm_inactive_count());
|
|
|
|
db_printf("vm_cnt.v_active_count: %d\n", vm_active_count());
|
|
|
|
db_printf("vm_cnt.v_laundry_count: %d\n", vm_laundry_count());
|
2018-02-12 22:53:00 +00:00
|
|
|
db_printf("vm_cnt.v_wire_count: %d\n", vm_wire_count());
|
2014-03-22 10:26:09 +00:00
|
|
|
db_printf("vm_cnt.v_free_reserved: %d\n", vm_cnt.v_free_reserved);
|
|
|
|
db_printf("vm_cnt.v_free_min: %d\n", vm_cnt.v_free_min);
|
|
|
|
db_printf("vm_cnt.v_free_target: %d\n", vm_cnt.v_free_target);
|
|
|
|
db_printf("vm_cnt.v_inactive_target: %d\n", vm_cnt.v_inactive_target);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1996-09-08 20:44:49 +00:00
|
|
|
|
2022-07-18 21:21:38 +00:00
|
|
|
DB_SHOW_COMMAND_FLAGS(pageq, vm_page_print_pageq_info, DB_CMD_MEMSAFE)
|
1996-09-08 20:44:49 +00:00
|
|
|
{
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
int dom;
|
|
|
|
|
2018-02-06 22:10:07 +00:00
|
|
|
db_printf("pq_free %d\n", vm_free_count());
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
for (dom = 0; dom < vm_ndomains; dom++) {
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
db_printf(
|
2017-01-03 00:05:44 +00:00
|
|
|
"dom %d page_cnt %d free %d pq_act %d pq_inact %d pq_laund %d pq_unsw %d\n",
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
dom,
|
|
|
|
vm_dom[dom].vmd_page_count,
|
|
|
|
vm_dom[dom].vmd_free_count,
|
|
|
|
vm_dom[dom].vmd_pagequeues[PQ_ACTIVE].pq_cnt,
|
Introduce a new page queue, PQ_LAUNDRY, for storing unreferenced, dirty
pages, specificially, dirty pages that have passed once through the inactive
queue. A new, dedicated thread is responsible for both deciding when to
launder pages and actually laundering them. The new policy uses the
relative sizes of the inactive and laundry queues to determine whether to
launder pages at a given point in time. In general, this leads to more
intelligent swapping behavior, since the laundry thread will avoid pageouts
when the marginal benefit of doing so is low. Previously, without a
dedicated queue for dirty pages, the page daemon didn't have the information
to determine whether pageout provides any benefit to the system. Thus, the
previous policy often resulted in small but steadily increasing amounts of
swap usage when the system is under memory pressure, even when the inactive
queue consisted mostly of clean pages. This change addresses that issue,
and also paves the way for some future virtual memory system improvements by
removing the last source of object-cached clean pages, i.e., PG_CACHE pages.
The new laundry thread sleeps while waiting for a request from the page
daemon thread(s). A request is raised by setting the variable
vm_laundry_request and waking the laundry thread. We request launderings
for two reasons: to try and balance the inactive and laundry queue sizes
("background laundering"), and to quickly make up for a shortage of free
pages and clean inactive pages ("shortfall laundering"). When background
laundering is requested, the laundry thread computes the number of page
daemon wakeups that have taken place since the last laundering. If this
number is large enough relative to the ratio of the laundry and (global)
inactive queue sizes, we will launder vm_background_launder_target pages at
vm_background_launder_rate KB/s. Otherwise, the laundry thread goes back
to sleep without doing any work. When scanning the laundry queue during
background laundering, reactivated pages are counted towards the laundry
thread's target.
In contrast, shortfall laundering is requested when an inactive queue scan
fails to meet its target. In this case, the laundry thread attempts to
launder enough pages to meet v_free_target within 0.5s, which is the
inactive queue scan period.
A laundry request can be latched while another is currently being
serviced. In particular, a shortfall request will immediately preempt a
background laundering.
This change also redefines the meaning of vm_cnt.v_reactivated and removes
the functions vm_page_cache() and vm_page_try_to_cache(). The new meaning
of vm_cnt.v_reactivated now better reflects its name. It represents the
number of inactive or laundry pages that are returned to the active queue
on account of a reference.
In collaboration with: markj
Reviewed by: kib
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8302
2016-11-09 18:48:37 +00:00
|
|
|
vm_dom[dom].vmd_pagequeues[PQ_INACTIVE].pq_cnt,
|
2017-01-03 00:05:44 +00:00
|
|
|
vm_dom[dom].vmd_pagequeues[PQ_LAUNDRY].pq_cnt,
|
|
|
|
vm_dom[dom].vmd_pagequeues[PQ_UNSWAPPABLE].pq_cnt);
|
Split the pagequeues per NUMA domains, and split pageademon process
into threads each processing queue in a single domain. The structure
of the pagedaemons and queues is kept intact, most of the changes come
from the need for code to find an owning page queue for given page,
calculated from the segment containing the page.
The tie between NUMA domain and pagedaemon thread/pagequeue split is
rather arbitrary, the multithreaded daemon could be allowed for the
single-domain machines, or one domain might be split into several page
domains, to further increase concurrency.
Right now, each pagedaemon thread tries to reach the global target,
precalculated at the start of the pass. This is not optimal, since it
could cause excessive page deactivation and freeing. The code should
be changed to re-check the global page deficit state in the loop after
some number of iterations.
The pagedaemons reach the quorum before starting the OOM, since one
thread inability to meet the target is normal for split queues. Only
when all pagedaemons fail to produce enough reusable pages, OOM is
started by single selected thread.
Launder is modified to take into account the segments layout with
regard to the region for which cleaning is performed.
Based on the preliminary patch by jeff, sponsored by EMC / Isilon
Storage Division.
Reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
2013-08-07 16:36:38 +00:00
|
|
|
}
|
1996-09-08 20:44:49 +00:00
|
|
|
}
|
2013-05-21 11:04:00 +00:00
|
|
|
|
|
|
|
DB_SHOW_COMMAND(pginfo, vm_page_print_pginfo)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
2018-12-30 15:58:18 +00:00
|
|
|
boolean_t phys, virt;
|
2013-05-21 11:04:00 +00:00
|
|
|
|
|
|
|
if (!have_addr) {
|
|
|
|
db_printf("show pginfo addr\n");
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
phys = strchr(modif, 'p') != NULL;
|
2018-12-30 15:58:18 +00:00
|
|
|
virt = strchr(modif, 'v') != NULL;
|
|
|
|
if (virt)
|
|
|
|
m = PHYS_TO_VM_PAGE(pmap_kextract(addr));
|
|
|
|
else if (phys)
|
2013-05-21 11:04:00 +00:00
|
|
|
m = PHYS_TO_VM_PAGE(addr);
|
|
|
|
else
|
|
|
|
m = (vm_page_t)addr;
|
|
|
|
db_printf(
|
2020-03-28 12:21:52 +00:00
|
|
|
"page %p obj %p pidx 0x%jx phys 0x%jx q %d ref 0x%x\n"
|
2013-08-09 11:11:11 +00:00
|
|
|
" af 0x%x of 0x%x f 0x%x act %d busy %x valid 0x%x dirty 0x%x\n",
|
2013-05-21 11:04:00 +00:00
|
|
|
m, m->object, (uintmax_t)m->pindex, (uintmax_t)m->phys_addr,
|
2019-12-10 18:14:50 +00:00
|
|
|
m->a.queue, m->ref_count, m->a.flags, m->oflags,
|
|
|
|
m->flags, m->a.act_count, m->busy_lock, m->valid, m->dirty);
|
2013-05-21 11:04:00 +00:00
|
|
|
}
|
1996-09-14 11:54:59 +00:00
|
|
|
#endif /* DDB */
|